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Abstract 


Introduction  or  background:  This  report  represents  Task  2  of  the  work  stream  “Research  Using 
in  Vivo  Simulation  of  Meta-Organizational  Shared  Decision  Making  (SDM)”,  one  component  of 
the  Technology  Innovation  Fund  (TIF)  program  on  Meta-organizational  Collaboration  t  hat  has 
been  designed  to  assist  in  understandi  ng  challenges  faced  by  the  Canadian  Forces  (CF)  .  The 
objective  of  the  stream  is  to  conduct  basic  research  into  shared  decision  m  aking  through  the 
analysis  of  case  studies,  exercises  and  simulations. 

Method:  This  task2  involved  the  developm ent  of  an  experimental  plan  to  si  mulate  an  in  vivo 
decision  making  environment  of  emergency  management  operations  and  to  c  ollect  data  on  the 
problem  solving  processes  and  outcomes  of  participants  in  the  simulation. 

Results  and  Discussion:  Based  on  the  Model  for  Inter-organizational  Problem  Solving 
developed  in  Task  1,  the  experimental  plan  presents  several  research  questions  related  to  the 
impact  that  intra-organizational  environment  has  on  the  ability  to  problem  solve  collaboratively, 
and,  to  the  impact  that  collaboration  has  on  improving  problem  solving  outcomes  and  processes. 
The  plan  outlines  two  research  co  mponents:  1)  Qualitative  analyses  to  identify  features  and 
cognitive  structures/pattems  that  guide  decisions  about  extreme  events;  2)  In  vivo  simulation  of  a 
complex  event  using  a  mix  of  organization  types  and  participants. 


Resume 


Introduction  :  Ce  rapport  presente  la  Tache  2  du  projet «  Recherche  par  la  simulation  in-vivo  sur 
la  prise  de  decision  partagee  des  meta-organisations  »,  une  composante  du  programme  de 
recherche  sur  les  meta-organisations  finance  par  le  Fond  pour  l’innovation  technologique 
(Technology  Innovation  Fund  -  TIF),  mis  en  place  afm  d’ameliorer  la  comprehension  des  defis 
auxquels  font  face  les  Forces  canadiennes  (FC)  en  matiere  de  collaboration  inter- 
organisationnelle.  L’objectif  de  ce  projet  est  de  mener  une  recherche  fondamentale  sur  la  prise  de 
decision  partagee,  au  moyen  d’etudes  de  cas,  d’exercices  et  de  simulations. 

Methode  :  La  Tache  2  a  implique  le  developpement  d’un  plan  experimental  afm  de  simuler  un 
environnement  sur  la  prise  de  decision  partagee  in-vivo  des  operations  de  gestion  des  urgences  et 
pour  colliger  des  donnees  portant  sur  les  processus  de  resolution  de  problemes  et  les 
performances  des  participants  dans  cette  simulation. 

Resultats  et  discussion  :  Base  sur  le  Modele  pour  la  resolution  inter-organisationnelle  des 
problemes,  developpe  en  Tache  1,  le  plan  experimental  teste  des  questions  de  recherche  liees  a 
l’impact  que  F environnement  intra-organisationnel  a  sur  la  capacite  de  resoudre  des  problemes 
en  collaboration.  Le  plan  decrit  deux  composantes  de  recherche  :  1)  Des  analyses  qualitatives  afm 
d’identifier  des  caracteristiques  et  des  structures  ou  modeles  mentaux  qui  guident  les  decisions 
lors  d’evenements  extremes;  2)  La  simulation  in-vivo  d’un  evenement  extreme  en  utilisant  des 
configurations  particulieres  d’ environnement  selon  les  types  d’organisations  et  de  participants. 
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Executive  summary 


Research  Using  In  Vivo  Simulation  of  Meta-Organizational  Shared 
Decision  Making  (SDM)  -  Task  2:  Experimental  Plan 

Louise  Lemyre1  et  al.;  DRDC  CSS  CR  2011-31;  Defence  R&D  Canada  -  CSS. 

Introduction  or  background:  This  report  describes  the  experimental  research  plan  for  Task  2  of 
the  project  entitled  Research  Using  in  Vivo  Simulation  of  Meta-Organizational  Shared  Decision 
Making  (SDM),  one  of  the  seven  work  streams  of  the  Technology  Innovation  Fund  (TIF)  project 
that  has  been  designed  and  implemented  to  assist  in  understanding  and  addressing  the 
collaboration  challenges  f  aced  by  the  Canadian  Forces  (CF).  The  CF  has  become  increasingly 
involved  in  collaborating  with  various  non- traditional  partners  to  find  and  to  implement  solutions 
to  address  complex  problems.  Many  complex  problems  faced  do  not  have  solutions  that  can  be 
found  within  the  CF  doctrine  and  protocols.  This  gap  has  resu  lted  in  the  CF  working  with 
organizations  that  may  have  different  appro  aches  to  problem  solving  and  d  ecision  making. 
Cooperation  and  collaboration  present  various  cha  llenges  and  opportunit  ies  to  the  CF  such  as 
identifying  approaches  that  allow  for  deep,  broad  organization  to  organization  collaboration,  and 
integrating  aspects  of  organizations  while  still  maintaining  individual  organizational  autonomy. 

The  objective  of  this  specific  stream  is  to  c  onduct  basic  research  into  shared  decision  making 
through  the  analy  sis  of  c  ase  studies,  e  xercises  and  simulations.  In  particular  the  research  will 
provide  recommendations  for  shared  decision  making  within  the  context  of  the  Incident 
Command  System  (ICS).  The  research  approach  will  utilize  a  simulation  system  to  examine  the 
decision  making  processes  of  officials,  responders  and  planners  as  they  detect,  prepare  for  and 
react  to  security  threats  and  events. 

Method:  Headed  by  Dr.  Lemyre,  the  Gap-Sante  te  am  at  the  University  of  Ottawa  develop  ed  an 
experimental  plan  to  test  in  vivo  elements  of  the  Model  for  Inter-organizational  Problem  Solving 
created  under  Task  1  of  th  e  project.  The  experime  nt  aims  to  demonstrate  that  the  Model  when 
implemented  in  vivo  can  produce  improvements  in  problem  solving  processes  and  outcomes  such 
as  better  quality  decisions,  higher  levels  of  satis  faction  with  proble  m  solving  processes,  better 
time-to-satisfaction  ratio  and  m  ore  cohesive  multi-organization  groups.  The  research  strategy 
outlines  an  approach  for  conducting  the  study  that  includes  two  resear  eh  components  1) 
Qualitative  interviews  and  analyses;  and,  2)  In  vivo  simulation  of  shared  decision  making  in  a 
complex  event.  The  qualitative  interviews  and  analyses  are  meant  to  compliment  the  in  vivo 
simulation  component  and  will  be  integrated  into  the  overall  findings  of  the  experiment. 

Results:  The  research  is  designed  to  seek  answe  rs  on  the  im  pact  that  intra-organizational 
environment  has  on  the  ability  to  problem  solve  in  a  collaborative  manner  and  on  the  impact  that 
collaboration  has  on  the  quality  of  problem  solving  outcomes  and  processes.  For  the  qualitative 
interview  component,  approximately  20  interviews  are  planned  with  d  ecision  makers  from  a 
variety  of  organizations  involved  in  one  or  more  extreme  events.  Interviewees  will  be  drawn  from 


1  Dr.  Lemyre  is  the  McLaughlin  Research  Chair  on  Psychosocial  Aspects  of  Risk  and  Health,  Director  of 
GAP-Sante  at  the  Institute  of  Population  Health,  and  Professor  of  Psychology,  Faculty  of  Social  Science, 
University  of  Ottawa 
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organizations  that  adhere  to  an  Incident  Co  mmand  System  (ICS)  approach  to  planni  ng  and 
responding  to  extreme  events,  e.g.,  CF,  RCMP,  and  from  organizations  that  do  not,  e.g., 
community  organizations,  private  sector.  The  interviews  will  be  conduct  ed  using  a  semi- 
structured  interview  guide  with  as  et  of  co  re  questions  and  probes  aim  ed  to  solicit  past 
experiences  and  interpretations  of  the  decision  makers  at  various  stages  of  problem  solving  during 
an  identified  extrem  e  event.  Analy  tic  software  will  be  used  to  classify  ,  sort  and  arrange  the 
qualitative  information  so  that  trends  and  patterns  can  be  analysed. 

The  research  design  for  the  in  vivo  simulation  component  calls  for  the  development  of  two  sets  of 
assessment  tools  to  ensure  that  the  study  is  prop  erly  implemented.  One  tool  is  needed  to  ass  ess 
scenario  complexity  (i.e.,  simple,  complicated,  complex)  and  another  to  asses  s  problem  solving 
approach  (i.e.,  independent,  coordinated,  cooperative,  complex).  (The  conceptual  undeipinni ngs 
of  these  tool  s  can  be  found  in  t  he  Task  1  report.)  The  resear  eh  design  assumes  a  complex 
situation  (control  variable)  within  which  decisi  on  makers  have  to  undertake  a  num  her  of  tasks 
characterized  as  either  “coordination”  or  “collaboration”.  Thus  a  num  her  of  scenarios  will  be 
developed  and  assessed  fo  r  their  degree  of  co  mplexity,  with  the  most  appropriate  one  utilised  in 
the  experiment.  Similarly,  a  number  of  coordination  and  collaboration  tasks  will  be  developed  to 
initiate  the  problem  solving  process  during  the  id  entified  event.  These  w  ill  be  assessed  agai  nst  a 
set  of  criteria  that  defines  coordination  and  co  llaboration  behaviours  and  the  most  appropriate 
tasks  will  be  chosen. 

The  in  vivo  simulation  design  includes  two  independent  variables  -  type  of  multi-organizational 
problem  solving  (i.e.,  coordination,  collaboration);  and,  multi-organization  environment. 
Participants  will  be  drawn  from  three  types  of  organizations  (military,  ICS  non-military  such  as 
emergency  services,  and  n  on-ICS  such  as  non-governmental  organizations)  and  grouped  in  pods 
(rooms)  in  a  number  of  different  ways  for  a  session  of  the  experiment.  One  sessi  on  will  hold  nine 
participants  grouped  t  hree  to  a  pod.  The  experi  ment  envisions  ei  ght  sessions  in  total,  with  the 
initial  set  of  four  focusing  on  coordinat  ing  tasks  and  the  second  set  focusing  on  collaborat  ing 
tasks. 

A  complex  scenario  will  be  delivered  to  each  pod  via  multi-media  injects  from  a  control  room. 
The  inter-pod  and  intra-pod  interactions  for  each  session  of  the  experi  ment,  prompted  by  task 
instructions,  will  be  video  and  audio  recorded.  This  will  provide  the  source  data  for  m  easurement 
of  problem  solving  outcomes.  The  research  design  includes  two  sets  of  dependent  variables  -  one 
focusing  on  problem  solving  processes  and  task  or  group  cohesion,  in  eluding  individual, 
collective  and  panel  satisfaction  with  the  probl  em  solving  process;  another  focusing  on  problem 
solving  outcomes  such  as  decision  quality  and  level  of  agreement  on  outcome.  Various  measures 
will  be  deve  loped  for  each  dependent  variable  (e.g.,  self  reports,  participant  ratings,  panel 
assessment,  etc.).  Once  the  dependent  variable  measures  are  fma  lized,  an  analy  sis  plan  will  be 
created  to  guide  the  detailed  descriptive  and  quantitative  analyses.  Findings  will  be  documented 
and  integrated  with  the  findings  from  the  qualitative  analyses  and  interviews. 

Next  Steps:  The  next  task  of  the  project,  Using  In  Vivo  Simulation  of  Meta-Organizational 
Shared  Decision  Making,  is  the  implementation  of  the  experimental  plan  (Task  3).  Final  steps 
include  the  modelling  of  communication  and  decision  making  functions  (Task4),  and  the 
development  of  a  user  friendly  knowledge  tool  with  recommendations  (Task  5). 
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Recherche  sur  le  partage  de  decision  des  meta-organisations  en  utilisant 
la  simulation  in  vivo  -  Tache  2  :  Plan  experimental 

Louise  Lemyre2  et  al.;  DRDC  CSS  CR2011-31;  R  &  D  pour  la  defense  Canada  -  CSS;  . 

Introduction  :  Ce  rapport  presente  les  resultats  d  ecoulant  de  la  Tache  2  du  projet  int  itule 
Recherche  par  la  simulation  in-vivo  sur  la  pri  se  de  decision  partagee  des  meta-organisations,  un 
des  sept  chantiers  du  Fond  pour  l’innovation  technologique  (Technology  Innovation  Fund  -  TIF), 
un  programme  de  recherche  congu  et  mis  en  place  afm  d’ameliorer  la  comprehension  des  defis 
auxquels  font  face  les  Forces  canadiennes  (FC)  en  matiere  de  collaboration  i  nter- 

organisationnelle.  Les  FC  collaborent  de  plus  en  plus  avec  de  nom  breux  partenaires  non 
traditionnels  afm  de  trouver  et  d  e  mettre  en  place  des  solutions  pour  faire  face  aux  problem  es 
complexes.  Les  solutions  a  plusieurs  d  es  problemes  complexes  rencontres  ne  se  trouvent  pas 
parmi  les  doctrines  et  les  p  rotocoles  des  FC.  Par  consequent,  cela  incite  les  FC  a  travailler  avec 
des  organisations  qui  utili  sent  differentes  approche  s  quant  a  la  resolution  de  problemes  et  a  1  a 
prise  de  decisions.  La  cooperation  et  la  co  llaboration  presentent  de  nom  breux  defis  et 
opportunites  pour  les  FC,  tels  que  l’identification  des  approches  qui  permettent  d’approfondir  et 
d’elargir  F organisation  de  la  collaboration  entre  les  organisations  et  d’  integrer  des  aspects  des 
organisations  tout  en  maintenant  l’autonomie  de  chacun. 

L’objectif  de  ce  champ  specifique  est  de  mener  une  recherche  fondamentale  portant  sur  la  prise 
de  decision  partagee  au  moyen  d’etudes  de  cas,  d’  exercices  et  de  simulations.  Plus 
particulierement,  cette  recherche  vise  a  foumir  des  recommandations  quant  a  la  prise  de  decision 
partagee,  dans  un  contexte  de  systeme  de  commandement  des  interventions  (SCI).  Le  systeme  de 
simulation  sera  F  approche  utilisee  pour  examiner  les  processus  de  prise  de  deci  sion  partagee  des 
autorites,  des  repondants  et  des  responsables  de  la  planification,  a  mesure  que  ceux-ci  detectent, 
preparent  et  reagissent  aux  menaces  a  la  securite  et  aux  evenements. 

Methode  :  Dirigee  par  1  a  Professeure  Lemyre,  Fequipe  Gap-Sante  de  F  Universite  d’Ottawa  a 
developpe  un  plan  experimental  pour  tester  les  elements  in  vivo  du  Modele  pour  la  resolution 
inter-organisationnelle  des  problemes,  developpe  1  ors  de  la  Tache  1.  L’experience  vise  a 
demontrer  que  le  Modele,  lorsqu’il  est  mis  en  oeuvre  in  vivo,  peut  produire  des  ameliorations 
quant  au  processus  de  la  resolution  de  problemes  et  ses  resultats,  tels  qu’une  meilleure  qualite  de 
decisions,  un  plus  haut  niveau  de  satisfaction  quant  au  processus  de  la  resolution  de  problemes, 
un  meilleur  rapport  temps-a-satisfaction,  et  d  es  groupes  m  ulti-organisationnels  plus  unis.  La 
strategic  de  recherche  presente  une  approche  pour  la  realisation  d’une  etude  incluant  deux  vo  lets 
de  recherche  :  1)  des  entre  vues  et  analyses  qualitatives  et;  2)  la  sim  ulation  in  vivo  de  la  prise  de 
decision  pendant  un  evenement  complexe.  Les  entrevues  et  analyses  qualitatives  ont  pour  but  de 
complementer  la  co  mposante  de  si  mulation  in  vivo  et  seront  integrees  dans  F  ensemble  des 
resultats  de  F  experience. 


2  Dre.  Lemyre  est  la  titulaire  de  la  Chaire  de  recherche  McLaughlin  sur  le  risque  psychosocial,  Directrice  de 
GAP-Sante  a  l’lnstitut  de  sante  des  populations,  et  Professeure  a  l’Ecole  de  psychologie,  Faculte  des 
sciences  sociales,  Universite  d’Ottawa. 
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Resultats  :  La  recherche  vise  a  mieux  comprendre  l’impact  que  l’envi  ronnement  inter- 
organisationnel  a  sur  la  capacite  de  resoudre  des  problemes  d’une  fag  on  collaborative  ainsi  que 
l’impact  que  la  collaboration  a  sur  la  qualite  des  resultats  et  les  processus  de  la  resolution  de 
problemes.  Quant  a  la  com  posante  de  l’entrevue  qualitative,  20  entretiens  sont  prevus  avec  les 
decideurs  d’une  variete  d’organisations  impliquees  dans  un  ou  plusieurs  even  ements  extremes. 
Les  personnes  interrogees  proviendront  des  organisations  adherant  a  une  approche  de  Systeme  de 
commandement  des  interventions  (SCI)  dans  la  pi  anification  et  la  reponse  face  a  des  evenements 
extremes  (par  exemple,  les  FC,  la  GRC)  et  des  organisations  que  n’y  adherent  pas  (par  exemple, 
des  organisations  communautaires,  le  secteur  pr  ive).  Les  entrevues  seront  effectuees  enutilisant 
un  guide  d’  entrevue  semi-structuree  avec  une  ser  ie  de  questions  genera  les  et  des  questions 
d’approfondissement  dans  le  but  de  sollicker  des  experien  ces  passees  et  les  interpretation  des 
decideurs  a  differents  stad  es  de  la  re  solution  de  problemes  lors  d’un  evenement  extre  me  donne. 
Des  logiciels  d’analyse  seront  utilises  pour  classi  fier,  trier,  et  organiser  1  es  informations 
qualitatives  afm  d’ analyser  les  tendances  et  les  modeles  emergeants. 

La  conception  de  la  recherche  pour  la  composante  de  simulation  in  vivo  necessite  l'elaboration  de 
deux  ensembles  d'outils  devaluation  pour  s'  assurer  que  l'etude  soit  correctement  m  ise  en  oeuvre. 
Un  outil  est  necessaire  pour  evaluer  la  complexite  des  scenarios  (p.ex.,  si  mple,  complique, 
complexe)  et  un  autre  pour  evaluer  l'approche  de  la  resolution  de  problemes  (p.ex.,  independante, 
coordonnee,  cooperative,  co  mplexe).  (Les  fonde  ments  conceptuels  de  ces  o  utils  peuvent  etre 
trouves  dans  le  rapport  d  e  la  Tache  1  .)  La  c  onception  de  la  recherche  suppose  une  situation 
complexe  (la  variable  controle)  au  sein  de  la  quelle  les  decideurs  ont  a  entreprendre  un  certain 
nombre  de  taches  caracterisees,  soit  par  la  «  coordination  »  ou  par  la  «  collaboration  ».  Ainsi,  un 
certain  nombre  de  scenarios  seront  elabores  et  evalues  quant  a  leur  degre  de  complexite,  et  le  plus 
approprie  sera  utilise  dans  l'experience.  De  meme,  un  certain  nombre  de  taches  de  coordination  et 
de  collaboration  seront  elaborees  pour  engager  le  processus  de  la  prise  de  decision  au  cour  s  de 
l'evenement  identifie.  Celles-ci  s  eront  evaluees  en  fonction  d'un  ense  mble  de  criteres  qui 
defmissent  la  coordination  et  la  col  laboration  des  co  mportements,  et  les  taches  les  plus 
appropriees  seront  choisies. 

Le  devis  de  la  sim  ulation  in  vivo  comporte  deux  variables  independantes  -  le  type  de  resolu  tion 
de  problemes  multi-organisationnels  (p.ex.,  la  coordination,  la  co  llaboration);  et  l'environnement 
multi-organisationnel.  Les  participants  seront  issus  de  trois  types  d'organisations  (l’armee,  les  SCI 
non-militaires  tels  les  services  d’  urgence  et  les  non-SCI  comme  les  groupes  non- 

gouvemementaux)  et  regroupes  en  equipe  (par  piece)  de  manieres  differentes  pour  chaque  session 
de  l'experience.  Une  session  comptera  neuf  participants  regroupes  en  trois  equipes.  L'  experience 
prevoit  huit  seances  au  tot  al,  dont  la  prem  iere  serie  de  quatre  portera  sur  la  coordination  et  la 
deuxieme  sera  axee  sur  la  collaboration. 

Un  scenario  complexe  sera  remis  a  chaque  equipe  via  une  assignation  m  ulti-media  a  partir  d'une 
salle  de  controle.  Les  inter  actions  inter-equipe  et  intra-equipe  guidees  par  des  instructions  pour 
chaque  session  de  l'experience,  seront  enregistr  ees  par  voies  video  et  audio.  Cela  constituer  a  la 
source  de  donnees  pour  L evaluation  des  resultats  de  la  resolution  de  problemes.  La  conception  de 
la  recherche  comprend  deux  ensembles  de  variables  dependantes  -  l'une  axee  sur  les  processus  de 
resolution  de  problemes  et  la  cohesion  de  la  tache  ou  du  groupe  incluant  la  satisfaction 
individuelle,  collective  et  d’un  panel  exteme  avec  le  processus  de  resolution  de  problemes,  l'autre 
mettant  l'accent  sur  la  resolution  de  problemes  t  els  que  la  qualite  des  resu  ltats  de  la  prise  de 
decisions  et  1  e  niveau  d'  accord  sur  le  r  esultat.  Diverses  mesures  seront  elaborees  pour  chaque 
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variable  dependante  (p.ex.,  les  mesures  auto-rapportees,  les  cotes  des  parti  cipants,  T  evaluation 
par  un  panel,  etc.)  Lorsq  ue  les  mesures  des  variables  dependantes  seront  f  inalisees,  un  plan 
d'analyse  sera  cree  afm  de  guider  1  'analyse  descriptive  detaillee  et  quantitati  ve.  Les  resultats 
seront  documentes  et  integres  aux  resultats  des  entrevues  qualitatives. 

Prochaines  etapes  :  La  prochaine  tache  du  projet,  I’Utilisation  de  la  simulation  in  vivo  de  la 
prise  des  decisions  meta-organisationnelles,  est  la  mise  a  ess  ai  in  vivo  du  cadre  experi  mental 
(Tache  3).  Les  dernieres  etapes  incluent  la  modelisation  des  fonctions  de  co  mmunication  et  de 
prise  de  decision  (Tache  4)  et  le  developpement  d’outils  de  connaissance  faciles  a  utiliser, 
accompagnes  de  recommandations  (Tache  5). 
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Preface 


This  document  represents  the  final  draft  of  Task  2:  Development  of  Experimental  Plan  for  In  Vivo 
Exercise  and  Simulation,  and  is  sub  mitted  using  th  e  DR  DC  supplied  template  for  Contractor 
reports.  The  work  has  been  completed  for  Defence  Research  and  Development  Canada  (DRDC)  as 
part  of  the  contract  deliverable  defined  in  the  project  entitled  Research  Using  In-Vivo  Simulation  of 
Meta-Organizational  Shared  Decision  Making  (SDM),  Contract  No.:  W771  4-083659/001/SV.  As 
per  contract  requirements,  the  document  is  provided  in  both  electronic  format  and  printed  copy  (5). 

The  document  presents  an  overview  of  the  resea  rch  strategy,  along  with  has  ic  assumptions  and 
guiding  principles  that  will  shape  the  co  nduct  of  the  proposed  study .  These  assumptions  will  need 
to  be  re-visited  for  currency  as  the  study  plan  is  implemented  in  Task  3  and  adjustments  made  to 
the  plan,  if  required.  The  plan  conforms  to  best  practice  with  its  i  nclusion  of  research  objectives 
and  descriptions  of  each  major  study  component,  along  with  the  methodologies  to  be  employed. 
The  plan  also  highlights  the  need  to  adhere  to  ethics  approval  processes  and  protocols  for  this  type 
of  research.  While  the  thrust  of  the  plan  as  deta  iled  will  remain  the  same,  some  specifics  regarding 
the  independent,  control  and  dependent  variables  may  change  as  further  discussions  ensue  about  the 
efficacy  and  efficiency  of  different  options  and  approaches. 

Document  Distribution  and  Confidentiality 

Document  distribution  and  confidentiality  protocols  as  specified  in  the  contract  noted  above  will 
apply  to  this  docum  ent.  Please  contact  Dr.  Louise  Lemyre,  Principal  Investigator,  University  of 
Ottawa,  at  louise.lemyre@uOttawa.ca  should  a  change  in  protocols  be  requested.  Currently,  this 
document  is  for  the  purview  of  the  Technical  Authority:  DRDC  and  government  only.  Please 
quote  with  due  reference  to  Lemyre  et  al.  2010,  Report  on  Research  Using  in  Vivo  Sim  ulation  of 
Meta-Organizational  Shared  Decision  Making  (SDM)  -  Task  2:  Experimental  Plan. 
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1  Introduction 


1.1  Overview  of  the  CORA  TIF  initiative 

As  outlined  in  an  overview  of  the  Centre  for  Operational  Research  and  An  alysis  (CORA)  and 
Technology  Innovation  Fund  (TIF)  of  Defence  Resear  eh  and  Develop  ment  Canada  (DRDC) 
(Chouinard,  2009),  the  C  anadian  Forces  (CF)  is  fi  nding  that,  as  an  organization,  it  is  becoming 
increasingly  involved  with  problems  that  defy  easy  solutions.  Sometimes  defined  as  “wicked 
problems”3,  these  problems  do  not  have  solutions  that  can  be  found  within  the  defined 
organizational  doctrine  or  protocols  of  the  CF.  Gi  ven  its  role  in  assisting  with  these  problems,  the 
CF  is  increasingly  working  with  numerous  non-traditional  partners.  The  CF  an  d  its  partners  find 
that  they  must  not  only  cooperate  and  coordinate  their  activities,  but  that  they  must  also  collaborate 
in  finding  and  contributing  to  suitable  solutions.  Often  this  coll  aboration  requires  deep,  broad 
organization  to  organization  collaboration.  This  level  of  collabor  ation  presents  various  challenges 
(and  opportunities)  such  as  the  integration  of  organizations  while  still  m  aintaining  individual 
organizational  autonomy.  The  CORA  TIF  project  ,  headed  by  Dr.  Paul  Chouinard  ,  has  been 
designed  and  implemented  to  assist  in  understanding  and  addressing  these  collaboration  challenges. 

1.1.1  CORA  TIF  project  goal  and  work  streams 

The  overall  goal  of  the  CORA  TIF  project  is  to: 

“assist  the  Canadian  Forces  (CF)  and  partnering  agencies  through  an  understanding  of  inter-agency 
collaborative  behaviour,  the  effects  of  inter-agency  relationships  on  collective  decision  making  and 
the  influences  of  psycho-social  factors.  In  addition,  the  research  will  seek  to  identify  mechanisms 
for  overcoming  social  and  cognitive  barriers  to  effective  collaboration.”  (Chouinard,  2009,  p.  2) 

To  meet  this  goal,  there  are  seven  work  strea  ms  that  are  being  implem  ented,  one  of  which  i  s  the 
stream  labelled  Using  In  Vivo  Simulation  of  Meta-Organizational  Shared  Decision  Making.  These 
are  listed  below. 

1.  Using  In  Vivo  Simulation  of  Meta-Organizational  Shared  Decision  Making 

2.  Improving  Geographical  Information  Exchange  through  an  Understanding  of  Agency  Specific 
Cognition 

3.  Modelling  Public  Safety  and  Security  Operations 

4.  Public  Safety  and  Security  Decision  Making  and  Role  of  Professionalism 

5.  Applying  Institutional  Analysis  to  the  Public  Safety  and  Security  Domain 

6.  Applying  Economic  Analysis  to  the  Meta-Organization 

7.  Multi-Level  Cultural  Influences  on  Decision  Making 


3  Wicked  problems  was  a  term  originally  coined  in  the  late  1960’s  in  operations  research  (Churchman,  1967), 
and  early  1970’s  to  describe  problems  related  to  social  policy  planning  (Rittel  &  Webber,  1973).  More 
recently,  the  term  has  been  broadened  to  describe  complex  situations  outside  of  planning  and  policy 
(Conklin,  2005). 
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1.1.2  Gap-Sante  project  goal  and  work  stream 

The  objective  of  the  work  stream  Using  In  Vivo  Simulation  of  Meta-Organizational  Shared 
Decision  Making  is  to  conduct  basic  research  into  shar  ed  decision  making  through  the  analysis  of 
case  studies,  exercises  and  simulations.  In  partic  ular,  the  research  will  provide  reco  mmendations 
for  shared  decision  making  within  the  contex  t  of  the  Incident  Command  Sy  stem  (ICS).  The 
research  approach  will  utilize  a  si  mulation  system  to  examine  the  decision  making  processes  of 
officials,  responders  and  planners  as  t  hey  detect,  prepare  for  and  react  to  security  threats  and 
events. 

1.2  Overview  of  project  tasks 

Within  the  GAP-Sante  pr  oject  work  stream,  there  are  five  tasks.  Each  task  is  briefly  described 
below. 

1.2.1  Task  1:  Synthesis  of  case  studies  to  form  a  shared  decision  making  (SDM) 
framework 

This  initial  task  entailed  com  pleting  two  independent  lines  of  investigati  on.  One  line  focused  on 
reviewing  various  fields  of  literature  including  shared  decision  m  aking  and  meta-organizati  onal 
theories.  The  other  line  focused  on  developing  case  studies  of  inter-organizational  decision  making 
processes  around  extreme  events  that  have  occu  rred  recently  in  Canada,  and  m  aking  some 
comparisons  with  international  cases.  The  findings  from  these  two  lines  were  then  integrated  into  a 
framework  of  meta-organizational  shared  decision  making.  The  SDM  framework,  which  is  labelled 
a  Model  for  Inter-Organizational  Problem  Solving,  serves  as  the  foundation  for  the  future  phases  of 
the  research.  Refer  to  report  entitled,  Research  Using  In  Vivo  Simulation  of  Meta-organizational 
Shared  Decision  Making  (SDM)  Task  1:  Synthesis  of  Case  Studies  to  form  a  SDM  Framework , 
(Lemyre  et  al.,  2009)  for  details. 

1.2.2  Task  2:  Development  of  an  experimental  plan  for  in  vivo  exercise  and 
simulation 

This  task  involved  the  development  of  an  experimental  plan,  as  presented  in  this  report,  to  simulate 
an  in  vivo  decision  making  environment  of  emergency  management  operations  and  to  collect  data 
on  the  problem  solving  processes  and  outcomes  of  participants  in  the  simulation.  The  experimental 
plan  is  designed  for  an  in  vivo  decision  making  environment  that  is  of  a  quality  that  is  equal  to  or 
better  than  research  done  via  the  current  Hydra  simulation  system.  Refer  to  details  on  the  Hydra 
system  at  the  following  site:  http://www.liv.ac.uk/Psvchology/ccir/hvdra.htmL 

1.2.3  Task  3:  Testing  the  shared  decision  making  framework  in  vivo 

From  the  results  of  Task  1,  identify  scenarios  that  challenge  the  SDM  framework,  as  well  as  the 
ICS  framework,  if  the  latter  differs  from  the  SD  M  framework.  These  scenarios  will  be  exercised 
within  the  simulated  environment  developed  in  Task  2.  Participants  in  the  simulations  will  include 
individuals  from  multiple  levels  of  g  ovemment,  multiple  jurisdictions  and  multiple  disciplines 
according  to  the  requirements  of  the  particular  scenario  being  exercised. 
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1.2.4  Task  4:  Modeling  communication  and  decision  making  functions 

For  Task  4,  given  the  results  of  the  laborat  ory  findings,  develop  a  model  of  the  em  ergency 
management  decision  functions  and  the  communications  functions  to  support  the  decision  function. 
The  model  of  these  functions  m  ust  describe  li  mitations  and  var  iations  based  upon  sit  uational 
factors  described  in  the  case  studi  es.  These  situational  factors  must  include  circumstances  where 
there  is  the  potential  for  failure  (e.g.,  the  failu  re  of  the  co  mmunications  infrastructure  during 
Flurricane  Katrina).  The  functions  must  also  be  related  to  the  SDM  and  ICS  frameworks  as  well  as 
activities  described  as  elements  of  capabilities  with  in  the  National  Incident  Management  System 
(NIMS). 

1.2.5  Task  5:  Developing  a  user  friendly  knowledge  tool 

Provide  guidance  for  the  operational  and  the  sup  porting  analytic  communities  through  a  software- 
based,  user  friendly  knowledge  database  tool  based  upon  MS  Access.  The  tool  must  provide  these 
communities  with  access  to  the  SDM  fra  mework  recommendations  for  shared  decision  m  aking, 
where  there  are  differences  between  the  SDM  framework  and  the  ICS  framework,  insight  into  the 
limitations  of  these  frameworks,  suggestions  to  overcome  the  limitations  in  what  circumstances  and 
any  other  insights  that  result  from  the  research. 


1.3  Purpose  of  present  report 

This  report  represents  a  draft  of  the  work  conducted  under  Task  2  of  the  project  entitled  Research 
Using  in  Vivo  Simulation  of  Meta-Organizational  Shared  Decision  Making  (SDM)  (Contract  No.: 
W7714-083659/001/SV).  It  details  the  experimental  plan  for  the  in  vivo  exercise  and  simulation  to 
be  implemented  under  Task  3. 

1.4  Overview  of  report  sections 

This  document  is  organized  into  seven  main  sections  with  various  sub-sections.  The  content  for 
each  of  these  sections  is  briefly  described  below. 


SECTION  1  Introduction 


•  This  section  outlines  the  background  of  the  CO  RA  TIF  project,  its  overarching  goal  and 
associated  work  streams.  The  objective  of  the  work  stream  Using  In  Vivo  Simulation  of  Meta- 
Organizational  Shared  Decision  Making  is  stated  and  each  of  the  five  tasks  associated  with 
completing  the  work  stream  is  described.  The  purpose  of  the  document  is  stat  ed  along  with  a 
brief  introduction  to  the  content  of  the  Task  2  report. 


SECTION  2  Overview  of  Research  Strategy 

•  This  section  begins  with  a  number  of  research  questions  related  to  impacts  that  approaches  to 
problem  solving  and  m  ulti-organizational  environments  have  on  problem  solving  processes 
and  outcomes.  It  briefly  describes  the  outcomes  of  Task  1  and  their  integration  into  the 
proposed  experimental  design.  An  approach  to  the  study  is  presented  and  details  on  the  two 
main  components  -  Qualitative  Interviews  and  In  Vivo  Simulation  Experiment.  The  section 
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concludes  with  a  brief  discussion  on  the  integration  and  synthesis  process  of  the  findings  from 
the  two  research  components. 


SECTION  3  Methods  for  Component  1 :  Qualitative  Interviews 

•  This  section  presents  a  rationale  for  the  qualita  tive  interview  com  ponent  of  the  study  and 
highlights  a  number  of  strengths  and  challenges  related  to  the  proposed  methodology. 

•  Details  are  provided  on  the  selection  process  of  extre  me  events,  and  on  the  selection  and 
recruitment  of  interviewees. 

•  The  interview  process  is  described  and  an  overview  of  the  planned  analysis  is  presented. 


SECTION  4  Methods  for  Component  2:  In  Vivo  Simulation  Experiment 

•  This  section  provides  an  overview  of  the  proposed  experiment  in  terms  of  the  choices  made 
re:  control,  independent  and  dependent  variables  and  type  of  study  environment.  A  sample 
profile  of  a  typical  participant  is  provided  to  assist  in  the  iden  tification  and  selection  of  study 
participants. 

•  A  key  control  variable,  i.e  .,  situation  complexity,  is  discussed.  Details  are  presented  on  the 
scenario  development  process  utilizing  factors  that  contribute  to  situation  com  plexity,  as 
outlined  in  the  Model  for  Inter-organizational  Problem  Solving.  Methods  used  to  assess  and 
validate  the  level  of  complexity  of  an  event  are  presented. 

•  Independent  variables  are  discussed  as  they  relate  to  ty  pe  of  multi-organizational  proble  m 
solving  and  multi-organization  environment.  Design  principles  for  the  problem  solving  tasks 
related  to  co  ordination  and  collaboration  are  presented,  along  with  descriptions  of  their 
development  and  assessment  processes.  Details  are  provided  on  how  members  of  different 
types  of  organizations  will  be  grouped  together  into  pods  to  simulate  the  multi-organizational 
environment.  Also,  session  design  details  are  discussed. 

•  Specifics  on  dependent  variables  are  discussed  related  to  both  problem  solving  processes  and 
outcomes.  Study  measures  for  the  dependent  variables  are  identified  and  discussed  in  terms  of 
their  validity,  reliability  and  ease  of  use. 

•  The  section  concludes  with  a  discussion  of  the  d  escriptive  and  inferential  research  techniques 
that  will  inform  the  data  analysis  plan. 


SECTION  5  Feasibility  and  Contingency  Analysis 

•  This  section  highli  ghts  a  series  of  issu  es  that  can  significantl  y  impact  the  conduct  of  the 
research  study.  These  issues  range  from  participant  availability  and  study  time  requirements 
to  technology  utilized  during  the  study  and  obtaining  ethics  approvals,  among  others. 
Contingencies  are  suggested  to  mitigate  these  identified  risks. 

SECTION  6  Implementation  Next  Steps 

•  This  section  provides  details  on  the  specific  tasks  to  be  perform  ed  in  implementing  both  the 
qualitative  interview  co  mponent  and  t  he  in  vi  vo  simulation  experiment  component  of  the 
study.  For  Component  1,  these  tasks  include  finalization  of  interview  questions  and  process; 
obtaining  ethics  approval;  recruitment  of  participants;  testing  of  instruments  and  process;  and 
launching  of  interviews.  For  Com  ponent  2,  details  cover  tasks  related  to  development  and 
calibration  of  scenarios  an  d  injects;  finalization  of  study  measures  and  protocols;  obtaining 
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ethics  approval;  control  ro  om  and  pod  equipment  set-up  and  testing;  trainin  g  of  technical 
staff/observers/facilitators;  and  recruitment  of  subjects.  A  project  schedule  i  s  provided  with 
proposed  start  and  end  dates  for  these  tasks. 


SECTION  7  Application  of  findings  and  future  considerations 

•  In  this  section  im  plications  are  raised  fo  r  training  collaborative  techniques  in  a  multi¬ 
organization  problem  solving  environment. 

•  This  section  also  suggests  opport  unities  for  extending  the  re  search  efforts  beyond  the 
experimental  phase  to  include  development  of  prototype  training  interventions  com  patible 
with  an  ICS  structure  and  the  testing  of  the  prot  otype  with  v  arious  target  groups.  T  he 
development  of  additional  scenarios  related  to  international  disasters  (e.g.,  Haiti),  “whole  of 
government”  events  and/or  community  level  collaborations  is  also  discussed. 
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2  Overview  of  Research  Strategy 


This  section  presents  an  overview  of  the  stra  tegy  that  has  been  developed  to  guide  the 
implementation  of  the  two  main  research  co  mponents  for  the  in  vivo  simulation  of  meta- 
organizational  shared  decision  m  aking.  Based  on  the  model  of  m  ulti-organizational  problem 
solving  developed  during  Task  1,  the  present  researc  h  strategy  outlines  an  approach  to  be  used  to 
better  understand  the  consequences  and  i  mpacts  that  implementing  this  type  of  model  can  have  on 
various  problem  solving  processes  and  outcomes.  Details  on  e  ach  of  the  specific  methods  are 
presented  in  Sections  3  and  4. 

2.1  Research  objectives  and  questions 

The  overall  research  objective  is  stated  as: 

To  better  understand  how  problem  solving  processes  and  outcomes  during  complex,  extreme  events 
can  be  impacted  by  1)  the  approaches  used  to  problem-solve,  and  2)  the  structure  and  governance 
of  the  multi-organizational  environment. 

Two  specific  research  questions  were  derived  from  the  overall  research  objective: 

Ql:  What  effect  does  the  approach  to  m  ulti-organizational  decision-making  have  on  problem 
solving  processes  and  outcomes? 

Q2:  What  effect  does  the  multi-organizational  compositional  environment  have  on  problem 
solving  processes  and  outcomes? 

2.2  Overview  of  strategy 

The  starting  point  for  the  strategy  is  the  Model  for  Inter-Organizational  Problem  Solving  that  was 
developed  as  part  of  Task  1  (see  Figure  1 ).  Within  the  iterative  cycle  of  phases  from  problem 
awareness  and  identification,  problem  definition,  solution  generation,  decision-making,  solution 
implementation,  feedback  and  solution  evaluation  (see  Figure  7  in  Annex  A),  the  model  outlined 
how  problem  solving  in  a  multi-organizational  environment  could  optimally  occur  taking  into 
account  1)  Situation  Complexity  (i.e.,  simple,  complicated,  complex)  (based  on  characterization  of 
the  situation  as  per  Figure  8  in  Annex  A),  2)  Timeframe  (from  pre-event  to  post-event)  ( Figure  9  in 
Annex  A),  3)  Asset  sharing  (power,  information,  resources  (money,  staff,  equipment))  {Figure  10  in 
Annex  A),  and  4)  Approa  eh  to  Problem  Solving  (i  .e.,  independence,  coordination,  coo peration, 
collaboration)  {Figures  11  to  14  in  Annex  A). 

As  outlined  in  the  research  objective  and  questions  above,  the  overall  goal  of  the  resear  eh  at  this 
stage  is  to  d  emonstrate  that  the  model  when  i  mplemented  in  vivo  can  produce  im  provements  in 
problem  solving  processes  and  outcomes  such  as  better  quality  decisions,  higher  levels  of 
satisfaction  with  problem  solving  process  es,  better  time-to-satisfaction  ratio  and  m  ore  cohesive 
multi- organization  groups. 
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Situation  Modifiers  Interorganizational 

Approach 


Figure  1:  Generic  model  of  inter-organizational  problem-solving  approaches  as  a  function  of 
situation  complexity,  assets  of  organizations  and  time  phase. 

As  illustrated  in  Figure  2  below,  there  will  be  two  main  research  co  mponents  which  will  be 
implemented  independently  from  one  another.  Component  1  focuses  on  improving  understanding 
of  the  cognitive  structures  and  patterns  (or  mind  maps)  that  decision  makers  use  to  guide  their 
actions  during  complex,  extreme  events.  This  will  consist  of  conducting  qualitative  semi-structured 
interviews  with  appr  oximately  20  individuals  who  have  held  decision  making  positions  during 
recent  extreme  events.  Component  2  consists  of  an  in  vivo  simulation  of  a  complex,  extreme  event 
using  scenarios,  injects,  pods  of  participants  and  detailed  tasks  that  have  been  designed  to  test  the 
extent  to  which  the  model  can  produce  better  pro  blem  solving  processe  s  and  outcomes.  The 
findings  from  Component  1  will  be  used  to  assist  in  analysing  and  interpreting  the  data  obt  ained 
from  Component  2. 
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Figure  2:  Overview  of  Research  Strategy > 

2.2.1  Component  1  -  Qualitative  interviews  with  decision  makers 

The  qualitative  interviews  are  designed  to  elicit  key  cognitive  structures  and  patterns  that  de  cision 
makers  use  to  guide  their  actions  during  extrem  e  events.  The  interviews  will  have  respondents 
reflect  on  various  stages  of  problem  solving  during  an  experienced  complex,  extreme  event  which 
involved  multiple  organizations.  Using  a  semi-structured  interview  guide,  the  interviewer  will  ask 
the  respondent  to  describe  their  vario  us  perceptions,  understanding  and  e  xperiences  of  how 
problem  solving  occurred  during  a  Ca  nadian  extreme  event  (e.g.,  Ice  Storm,  SARS),  and  their 
involvement  in  these  processes.  In  anal  ysing  the  transcripts  from  these  interviews,  the  team  will 
attempt  to  identify  common  cognitive  patterns  and  structures  used  by  respondents.  These  will  be 
compared  and  contrasted  across  different  t  ypes  of  respondents,  events,  stages  of  problem  solving, 
situation  complexity,  and  organization  t  ype.  These  semi-structured,  self-report  m  ethods  produce 
data  that  all  ow  the  rese  archer  to  as  sess  assumptions,  frameworks  and  m  odels  employed  by 
respondents  (Klein  et  al.,  1989).  The  analysis  of  interviews  can  provide  a  better  understanding  of 
context,  expertise,  and  tacit  knowledge  of  the  respondents  (Nja  &  Rake,  2009). 

The  interview  process  will  use  an  adapted  version  of  Critical  Decision  Method  (Klein  et  al.,  1989) 
as  outlined  by  Smith  and  Dowell  (2000).  Interviewers  will  be  guided  by  a  semi-structured  interview 
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guide  developed  specifically  for  this  component.  The  guide  will  contain  core  questions  designed  to 
elicit  responses  on  the  main  topic  areas  (e.g.,  probl  em  solving  approaches,  situation  com  plexity, 
decision  outcomes)  within  the  fra  mework  of  Critical  Decision  Method  which  focuses  on  are  as  of 
incident  identification,  ti  me-line  and  decision-  point  verification,  derivation  and  connection  of 
details,  and  alternative  exploration.  In  additi  on  to  the  core  questions,  follow-up  probes  will  be 
designed  to  encourage  the  respondent  to  provi  de  more  detail,  explanation,  or  interpretation,  as 
required  at  the  various  stages.  Overall  ,  it  anticip  ated  that  approxim  ately  20  interviews  will  be 
conducted  with  each  interview  lasting  approximately  one  hour. 

Secondary  source  material,  depending  on  availability,  might  also  be  considered  for  qualitative 
analysis.  Anonymized  transcripts  of  existing  interviews  and  n  arratives  on  decisions  may  be 
analyzed  using  the  grid  developed  for  Co  mponent  1  and,  in  doin  g  so,  enriching  the  infor  mation- 
base  with  other  historical  experiences  of  decision-making  during  major  events. 

2.2.2  Component  2  -  In  vivo  simulation  experiment 

The  second  research  component  will  consist  of  an  experimental  study  that  has  been  designed  to  test 
the  Model  for  Inter-organizational  Problem  Solving.  The  study  environment  will  consist  of  three  3- 
person  “pods”  connected  to  one  a  nother  and  to  a  co  ntrol  room  via  video  conferencing  equipment. 
From  the  control  room,  the  pods  will  be  introduced  to  an  evolving  complex,  major  event  scenario 
with  various  timed  injects,  and  specific  instructions  and  problem  solving  tasks  that  they  are  to 
undertake  both  within  and  between  pods  during  the  session.  The  independent  variables  that  will  be 
manipulated  during  the  experiment  include: 

•  Approach  to  problem  solving: 

♦  two  levels  of  approach  to  problem  solving  (coordinating,  collaborating); 

•  Multi-organizational  environment: 

♦  two  levels  of  pod  composition  (same  type  of  organization  -  h  omogeneous,  mixed 
types  of  organization);  and 

♦  two  levels  of  inter-pod  interaction  (no  interaction;  open  interaction) 

The  main  dependent  variables  will  be  problem  solving  processes  and  outcomes  such  as  decision 
quality,  satisfaction  with  problem  solving  process  a  ccording  to  a  ctors  and  to  external  panel,  task 
cohesion,  participation,  time  and  agreement. 

Pods  will  be  populated  with  individuals  who  have  experience  working  at  strategic  and  operational 
levels  in  planning  for  and  responding  to  extreme  events.  Subjects  will  be  recruited  from  various 
types  of  org  anizations  including  th  ose  that  follow  m  ore  of  an  incident  command  sy  stem  (ICS) 
approach  (e.g.,  military,  police,  emergency  services),  and  those  from  organizations  where  an  ICS 
approach  is  less  ty  pical  (e.g.,  public  health  ag  encies,  social  services,  financial  institutions, 
community  organizations).  Eight  sessions  will  be  i  mplemented,  with  nine  participants  per  session. 
Sessions  are  expected  to  be  approximately  four  ho  urs  in  duration  and  will  be  audio-  and  video- 
recorded. 

2.2.3  Integration  of  findings  from  components 

The  findings  from  the  interview  co  mponent  using  qualitative  methods  will  com  plement  the  work 
that  is  being  planned  for  the  in  vivo  experiment  component  that  relies  prim  arily  on  quantitative 
methods.  It  is  anticipated  that  by  qualitatively  understanding  the  key  cognitive  structures  and 
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patterns  that  decision  makers  use  to  guide  their  ac  tions  in  extreme  event  problem  solving,  the  team 
will  be  able  to  use  this  information  to  assist  in  analysing  and  interpreting  some  of  the  findings 
derived  from  the  in  vivo  experiment. 
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3  Methods  for  Component  1  -  Qualitative  Interviews 


3.1  Overview  of  design 

Qualitative  research  methods  enable  researchers  to  address  resear  eh  questions  in  a  manner  that  is 
often  complementary  to  using  quantitative  methods.  While  quantitative  methods  are  often  designed 
to  test  specific  hy  potheses  and  following  stri  ct  assumptions,  qualitative  methods  allow  the 
researcher  to  elicit  explanations,  discover  pa  tterns,  and  understand  re  lationships.  Qualitative 
methods  are  often  used  t  o  find  answers  that  g  o  beyond  the  scope  of  quantitative  methods  (Crotty, 
1998;  Smith,  2008).  For  the  present  study ,  findings  from  qualitative  interviews  will  be  used  to 
assist  in  the  analysis  and  interpretation  of  the  data  obtained  from  the  more  quantitative  in  vivo 
experimental  component. 

Interviews  with  key  stakeholders  are  a  retrospec  tive  method  that  can  be  used  to  study  the  decision 
making  processes  during  emergency  situations  (Nja  &  Rake,  2009).  For  the  present  study,  we  will 
focus  on  the  broader  process  of  proble  m  solving  which  includes  decision  making  as  one  of  the  key 
stages.  The  overall  approach  for  interviewing  in  the  present  study  has  been  adapted  from  the 
Critical  Decision  Method  approach  to  interviewing  (Klein,  Calderwood  &  MacGregor,  1989) 
which  has  been  simplified  and  adapted  by  other  authors  (e.g,  Smith  and  Dowell,  2000;  Nja  &  Rake, 
2009).  Using  a  semi- structured  interview  guide  with  core  questions  and  probes,  the  interviewer  will 
work  with  the  decision  makers  to  elicit  past  experiences  and  interpretations  at  various  stages  of 
problem  solving  during  an  identified  extreme  event. 

The  main  strengths  of  using  this  type  of  approach  have  been  identified  as: 

•  The  analysis  of  qualitative  data  collected  through  semi-structured  interviews  can  provide 
novel  insights,  perspectives  and  understanding  of  key  constructs  or  concepts  (Richards  & 
Morse,  2007); 

•  Interviews  with  key  decision  makers  can  provide  important  information  on  context, 
expertise,  and  tacit  knowledge  required  to  effectively  operate  in  extreme  events  that  can  be 
applied  to  multiple  systems,  settings  and  contexts  (Nja  &  Rake,  2009;  Klein  et  al.,  1989); 
and 

•  Qualitative  semi-structured  interviews  can  be  used  to  understand  the  cognitive  structures 
and  processes  that  decision  makers  use  to  guide  their  actions  during  extreme  events  (Klein 
etal,  1989). 

Some  of  the  key  challenges  to  using  this  approach  which  have  been  considered  in  the  development 
of  the  methods  for  this  study  include: 

•  Reliability  of  a  posteriori  accounts  from  those  involved  in  events  including  retrospective 
distortions  and  faulty  memories  (Nja  &  Rake,  2009;  Latiers  &  Jacques,  2009); 

o  Our  design  will  focus  on  events  that  have  occurred  within  the  past  ten  years,  so  that 
the  recall  will  be  easier  than  with  events  further  in  the  past.  Our  emphasis  will  be 
on  the  interpretation  of  the  event  from  their  perspective  in  order  to  understand  the 
cognitive  structures  and  processes  they  likely  used  to  guide  their  actions  during  the 
event.  The  goal  is  not  to  have  a  “factual”  accounting  of  the  event,  but  rather,  an 
“interpreted”  accounting  of  the  event. 

•  Social  desirability  bias  where  respondents  are  more  likely  to  present  a  positive  image  of 
themselves  during  the  interview  (Fisher,  1993); 
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o  This  is  less  likely  to  be  a  concern  given  that  the  interviews  will  focus  on 

interpretations  of  the  event  rather  than  qualitative  judgements  on  appropriateness, 
quality  of  decisions,  etc. 

•  Response  burden  for  interviewees; 

o  Given  the  time  demands  on  the  respondents,  we  will  endeavour  to  limit  the 
interview  to  approximately  60  minutes,  and  will  travel  to  their  workplace  to 
conduct  the  interviews,  if  appropriate. 


3.2  Selection  of  events  and  interviewees 

Selection  of  extreme  events 

Three  to  four  Canadian  extreme  events  that  have  occurred  within  the  past  ten  years  will  be  selected 
from  the  case  studies  that  were  undertaken  in  Task  1  (e.g.,  SARS,  Ice  Storm).  Selection  of  events 
will  be  undertaken  in  collaboration  with  DRDC,  taking  into  account  various  considerations  such  as 
situation  complexity,  nature  of  event,  and  orga  nizations  involved,  as  well  as  pragmatic  concerns 
such  as  accessibility  and  location  of  potential  interviewees. 

Selection  of  interviewees 

Interviewees  will  be  select  ed  according  to  their  association  with  the  select  ed  extreme  events.  In 
order  to  be  considered  for  inclusion  in  the  st  udy,  potential  interviewees  need  to  have  had  the 
authority  to  make  key  decisions  on  behalf  of  an  organization  that  was  present  and  be  involved  in 
the  decision  making  process  during  the  selected  extreme  event.  We  anticipate  that  these  decision 
makers  will  be  an  excellent  source  of  information  regarding  how  situation  complexity,  approach  to 
problem  solving,  and  various  mediating  factors  can  impact  on  problem  solving  processes  and 
outcomes. 

The  goal  will  be  to  recruit  decision  makers  from  a  variety  of  organizations  that  were  inv  olved  in 
one  of  a  small  num  ber  of  extreme  events.  Within  each  event,  there  will  be  an  attempt  to  recruit 
some  decision  makers  from  organizations  that  rel  y  primarily  on  the  Incident  Command  Sy  stem 
(ICS)  approach  to  plannin  g  for  and  respondi  ng  to  extreme  events  (e.g.,  CF,  RCMP,  EMO),  and 
some  from  organizations  that  rely  less  on  ICS  approaches  (e.g.,  community  organizations,  health 
authorities,  private  sector). 

The  anticipated  num  ber  of  interviewees  required  is  around  20.  This  would  include  multiple 
interviewees  from  each  selected  event  representing  different  stak  eholders.  This  nu  mber  would 
provide  the  study  with  rich,  de  tailed  data  fro  m  interviewees  representing  different  event  s  and 
different  organizations,  and  permit  an  analysis  of  data  according  to  type  of  org  anization  and  event. 
A  common  standard  for  long,  qualitative  interviews  is  generally  up  to  10  people  (Creswell,  1998). 
The  rationale  for  substantially  increasing  this  num  ber  is  that  t  he  interviews  are  likely  to  be 
relatively  short  for  this  type  of  interviewing  (approximately  one  hour)  given  the  time  demands  on 
the  interviewees,  and  the  analysis  plan  requires  examining  patterns  within  and  across  sub-groups  of 
interviewees  (e.g.,  by  organization  type,  by  complexity  of  event). 

Recruitment  of  interviewees 

Recruitment  will  likely  be  challenging  given  the  timing  of  the  data  collection  (summer  and  fall),  the 
relatively  senior  level  of  interviewees,  and  the  number  of  interviews  required.  To  m  itigate  the 
potential  challenges,  a  flexible  recruitment  pro  cess  will  be  developed  w  ith  various  stages  and 
procedures  outlined  such  as  identifying  potential  interviewees,  making  initial  contact,  securing 
agreement  to  participate,  scheduling,  obtaining  informed  consent,  and  conducting  the  interview. 
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Another  avenue  to  gathe  r  useful  m  aterial  will  be  to  consider  existing  transcripts  of  r  elevant 
interviews  and  narratives  about  major  events  an  d  decision-making.  Given  that  the  aim  of  a 
qualitative  strategy  is  not  to  collect  information  in  a  systematic  fashion  but  to  gather  insight  into 
interpretation  that  will  shed  new  light  onto  processes,  the  ad-h  oc  procedure  of  opportunistic 
analysis  of  existing  m  aterial  is  fully  legitimate.  Hence  we  could  use  existing  verbatims  of  key 
informants  about  significant  events  and  transferred  them  into  our  grid  of  analysis. 

3.3  Interview  process 

After  securing  ethics  approval  from  the  University  Tri-Council  Research  Ethics  Board,  interviews 
will  be  conducted  in-person  by  a  trained  interviewer  using  a  semi-structured  interview  guide.  Once 
informed  consent  has  been  achieve  d  after  the  interviewer  has  brie  fly  presented  the  purpose  of  the 
interview,  the  main  foci,  and  general  guidelines  for  the  interview  process  (e.g.,  there  are  no  right  or 
wrong  answers,  ask  for  clarifications  on  ques  tions),  the  process  will  begin  by  having  the 
interviewee  read  through  and  sign  a  consent  form.  The  interviewer  will  then  use  a  sem  i-structured 
interview  guide  to  ask  the  interviewee  a  number  of  open-ended  q  uestions  and  various  folio  w-up 
probes  that  will  be  designed  to  elicit  information  on  the  m  ain  topic  areas  (  e.g.,  problem  solving 
approaches,  situation  complexity,  decision  outcomes  and  processes)  (pleas  e  refer  to  Annex  B  for 
examples  of  potential  questions). 

The  overall  goal  of  the  interview  will  be  to  extract  and  understand  the  cognitive  structures  and 
processes,  and  schema,  which  guided  the  actions  of  the  decision  makers  through  various  stages  of 
problem  solving  during  the  extrem  e  event.  The  interview  will  be  an  adaptation  of  the  Critical 
Decision  Method  approach  to  intervie  wing  which  focuses  on  questions  that  bring  out  incident 
identification,  time-line  and  decision-  point  verification,  derivation  and  co  nnection  of  details 
(deepening),  and  alternative  exploration  (  what  if  queries).  It  is  an  ticipated  that  the  interview  will 
start  by  having  the  interviewee  provide  an  account  of  the  ev  ent  from  his/her  own  experience. 
During  the  a  ccount,  the  i  nterviewer  tracks  the  main  events,  situation  assessments,  and  decisions 
provided  by  the  interviewee.  The  interviewer  then  reviews  the  account  of  the  incident  with  the 
interviewee  and  asks  follow-up  questions  or  probes  to  elicit  information  on  the  main  topics  areas. 

The  interviews  will  last  o  n  average  60  m  inutes  and  be  conducted  at  a  location  convenient  for  the 
interviewee  (e.g.,  their  office).  All  interviews  will  be  audio-recorded  so  t  hat  they  can  be  fully 
transcribed. 

3.4  Analysis  plan 

The  audio  recordings  from  each  interview  will  be  fully  transcribed.  The  verbatim  transcripts  will  be 
imported  into  the  NVivo  program  prior  to  analysis.  NVivo  is  an  analytic  software  program  designed 
to  assist  with  classifying,  sorting  and  arranging  qualitative  information  so  that  trends  and  patterns 
can  be  analysed. 

Once  the  transcripts  have  been  im  ported,  the  firs  t  step  of  the  analy  sis  will  involve  identi  fying 
sections  of  transcripts  that  are  rel  ated  to  the  pe  rceptions  of  situation  complexit  y  (i.e.,  simple, 
complicated,  complex).  These  sections  will  be  screened  and  separated  into  meaningful  units  of 
information.  The  next  s  tage  of  the  analysis  will  involve  a  nalysing  the  identified  units  of 
information  by  expected  themes  (e.g.,  m  ulti  organizational  environment,  approach  to  pr  oblem 
solving,  problem  solving  stage),  as  well  as  any  emerging  themes. 

The  analysis  will  rely  extensively  on  the  quality  of  data  coding  (i.e.,  assi  gning  text  or  units  of 
information  to  categories).  Data  coding  will  be  conducted  in  a  cascading  manner.  This  means  data 
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will  be  categorized  initially  into  a  small  number  of  broad  categories.  The  broad  categories  will  then 
be  coded  into  additional  levels  of  subcategories  as  increased  levels  of  specificity  are  required  to 
support  the  analysis  of  themes.  To  measure  the  reliability  of  the  coding  scheme  or  categorizing 
system,  once  established,  1 0%  of  the  interviews  wi  11  be  coded  by  two  differe  nt  researchers,  and 
then  the  coding  results  com  pared.  Where  consensus  does  not  exist,  these  aspects  of  the  coding 
scheme  will  be  re-examined  and  further  refined  or  adjusted. 
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4  Methods  for  Component  2  -  In  Vivo  Simulation 
Experiment 


4.1  Overview  of  experiment  design 

4.1.1  Control,  independent  and  dependent  variables 

The  first  stage  of  this  project,  Task  1:  Synthesis  of  Case  Studies  to  form  a  SDM  Framework, 
identified  a  number  of  key  variables  for  further  st  udy  with  respect  to  m  eta-organizational  shared 
decision  making.  The  control  variable  of  situation  complexity  was  selected  to  be  co  mplex,  i.e.  the 
experiment  will  focus  only  on  a  scenario  that  has  been  initially  assessed  as  complex.  Depending  on 
the  results  of  the  planned  stud  y,  future  work  may  focus  on  converting  situation  complexity  from  a 
control  variable  to  an  independent  variable  to  de  termine  the  extent  to  which  event  co  mplexity  has 
an  impact  on  the  problem  solving  processes.  The  two  independent  variables  used  in  the  experiment 
design  are  1)  approach  to  multi-organizational  decision-making  (coordination  and  collaboration), 
and  2)  t  ypes  of  multi-organizational  environments  (homogeneous  and  mixed  types  of 
organizations).  The  dependent  variables  will  include  various  measures  of  problem  solving 
processes  and  outcomes. 

4.1.2  Description  of  subjects/sample 

Characteristics  of  subjects 

Subjects  recruited  for  the  study  will  consist  of  decision  makers  who  would  normally  be  involved  in 
the  response  to  extreme  events  on  behalf  of  their  organizations.  They  will  be  recruited  from  various 
sectors  and  types  of  organizations  including  public,  private  and  non-governmental  organizations 
normally  involved  in  planning  for,  responding  to  and/or  managing  complex,  extreme  events  at  the 
municipal,  regional  or  national  level. 

It  is  anticipated  that  the  higher  the  level  of  decision  makers  in  the  sample,  the  more  persuasive  will 
be  the  re  suits.  Subjects  must  have  t  he  authority  and  experience  to  make  both  strategic  an  d 
operational  decisions  during  an  em  ergency  situation.  Therefore,  subject  recruitment  will  focus  on 
finding  appropriate  subjects  who  are  higher- level  decision  makers  and  who  fit  the  inclusion  / 
exclusion  criteria  outlined  in  Table  2.  As  senior  level  decision  makers,  these  participants  during 
extreme  events  would  normally  assume  roles  and  responsibilities  that  are  particular  to  their  level  of 
influence  and  expertise.  For  exam  pie,  military  captains  and  majors,  chiefs  and  coordi  nators  of 
emergency  services,  executive  officers  of  inter  ested  business  associations  and  non-govemmental 
organizations  would  all  make  ideal  candidates. 

In  order  to  replicate  the  convergence  that  occurs  on  the  scene  of  an  extreme  event,  subjects  will  be 
selected  from  different  sectors  and  organization  t  ypes.  The  part  icipants  will  be  coming  from  a 
number  of  sectors,  including  subjects  from  the  military,  other  organizations  that  use  employ  an  ICS 
structure,  and  organizations  that  use  altemativ  e  organizational  structures.  By  recruiting  subjects 
from  both  I  CS  and  n  on-ICS  organizations,  the  impact  of  orga  nizational  structures  on  d  ecision 
making  canbemeasur  ed.  Additionally,  the  study  of  subjects  fro  m  different  sector  s  and 
organization  types  makes  it  possible  to  measure  differences  in  problem  solving  am  ong  different 
workplace  cultures,  providing  further  insight  into  the  relationship  between  problem  solving  and 
multi- organizational  contexts. 
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One  step  in  finalizing  the  sampling  plan  will  be  to  identify  specific  key  organizations  from  which  to 
recruit  participants.  At  this  point  in  the  planning  ,  we  have  identified  six  ke  y  sectors,  and  examples 
of  the  types  of  organizations  that  could  be  approached  (  see  Table  below  for  examples).  The  specific 
organizations  selected  classification  of  organizatio  n  type  and  distribution  of  participants  across 
organizations  will  be  finalized  in  the  sampling  plan  to  be  discussed  with  CSS  as  part  of  Task  3. 


Table  1:  Key  sectors  and  example  organizations 


Sector/Type  of  Organization 

Example  Organizations 

Military 

•  Maritime  Command 

•  Land  Force  Command 

•  Air  Command 

Other  Federal  Government 
Departments  and  Agencies 

•  Canadian  Nuclear  Safety  Commission 

•  Public  Health  Agency  of  Canada 

•  Public  Safety 

•  Royal  Canadian  Mounted  Police 

•  Transport  Canada 

Provincial  Government 
Departments  and  Agencies 

•  Emergency  Measures  Organizations  (EMO’s) 

•  Health  ministries 

•  Provincial  police  forces 

•  Environment  ministries 

Regional  Government 

Departments  and  Agencies 

•  City  officials 

•  Emergency  operations  centres  (EOC’s) 

•  Fire  departments 

•  Police  departments 

Non-Governmental 

Organizations  and  Associations 

•  Canadian  Red  Cross 

•  Community  organizations 

•  St.  John  Ambulance 

•  The  Salvation  Army 

Private  Sector  Organizations  and 
Associations 

•  Business  associations 

•  Information  management  organizations 

Screening  criteria 

Once  the  specific  organizations  have  been  selected,  a  screening  process  will  be  undertaken  to 
ensure  that  the  individuals  invited  t  o  participate  from  each  organization  correspond  to  t  he 
recruitment  profile  of  higher-level  decision  makers  within  their  respective  organizations.  The 
selection  process  for  individual  subjects  will  rely  on  specific  criteria  in  order  to  ensure  that  subjects 
participating  have  similar  levels  of  decision-making  authority  in  their  respective  organizations,  and 
have  some  experience  with  planni  ng  for,  responding  to  an  d/or  managing  extreme  events.  The 
specific  inclusion  and  exclusion  criteria  for  screening  of  individuals  are  presented  in  Table  2. 
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Table  2:  Inclusion  and  exclusion  criteria  for  selecting  participants 


Inclusion  Criteria 


1 .  Previous  experience  in  both  strategic  and  operational  decision-making  during  at  least  one 
major  event  (not  necessarily  with  current  organization) 


2.  Have  been  in  a  management-level  decision  making  role  for  his/her  organization  for  at  least  12 
continuous  months 


3.  Has  the  authority  to  make  decisions  regarding  allocation  of  resources  (human,  financial  and 
equipment)  on  behalf  of  their  organization  (or  region)  with  respect  to  major  events 


4.  Able  and  willing  to  volunteer  to  participate  in  study 


Exclusion  Criteria 


5.  Outside  commuting  distance  of  a  study  site 


6.  Unable  to  effectively  communicate  in  the  official  language  of  the  study  session 


4,1.3  Description  of  study  environment  and  processes 

This  section  describes  the  study  environment,  information  flows  and  processes  needed  to  conduct 
the  experiment. 

4.1.3. 1  Facility  requirements 

A  control  room  and  three  separ  ate  rooms  for  th  e  pods  are  needed  to  run  each  session  of  the 
experiment.  The  control  room  will  be  used  to  transmit  the  initial  scenario  to  the  pods  and  to  provide 
instructional  tasks  and  informational  injects  to  a  dvance  the  si  mulation.  It  will  also  be  used  to 
observe  the  occupants  of  the  pods  and  to  audio-  and  video-record  the  events  within  each  pod.  Each 
pod  will  consist  of  three  st  udy  participants.  Refer  to  section  4.2.3  Multi-organization  environment 
for  detailed  information  on  individual  pod  composition.  The  control  room  and  pod  rooms  will  be  in 
one  physical  location. 

The  pod  rooms  will  be  id  entical  as  much  as  possible  in  terms  of  equipment  and  physical  layout. 
Each  pod  room  will  have  three  online  conferencing  st  ations,  web  cameras,  a  printer,  three  headsets 
with  earphones  and  microphones,  white  board,  three  eh  airs,  a  table  in  the  center  for  discussions, 
pens,  markers  and  other  a  ssorted  stationary.  There  will  be  a  m  onitor  either  mounted  to  the  wall 
directly  in  front  of  the  table  or  sitting  on  one  end  of  the  table.  This  monitor  will  be  linked  to  one  of 
the  computers  so  that  participants  will  be  able  to  maximize  their  use  of  the  conferencing  soft  ware, 
when  made  available  to  the  pods. 

The  control  room  will  serve  as  the  n  erve  centre  for  the  sessio  n.  In  addition  to  recording  and 
moderating  the  online  conference,  the  control  room  will  video  and  audio-record  the  participants  in 
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each  pod,  collect  the  data  generated  from  the  injects,  and  feed  all  the  critical  information  to  the  pod 
members,  i.e.,  the  scenario  injects  and  tasks.  The  control  room  will  also  run  the  online  conference 
for  an  initial  virtual  introductory  session  and  the  debriefing  at  the  end  of  the  simulation.  Contained 
inside  the  control  room  will  be  a  table,  conferencing  computer,  monitor,  headset  and  three  monitors 
to  observe  the  video  feeds. 

4.1.3.2  Information  flows  and  processes 

Study  subjects  will  be  able  to  comm  unicate  interpersonally  with  each  other  in  their  own  pod  given 
they  are  in  the  same  roo  m.  Conversations  betw  een  individuals  will  be  recorded  and  the  entire 
session  will  be  videotaped.  In  addition,  each  subj  ect  will  be  provided  information  and  the  tools 
needed  to  communicate  with  members  from  the  other  two  pods  in  the  sess  ion  during  s  pecific 
periods  of  the  session.  The  figure  below  illustrates  the  paths  of  communication  between  the  control 
room  and  the  pod  rooms.  It  also  illustrates  the  medium  used  to  capture  communication  between  and 
within  the  pod  rooms. 


Figure  3:  Communication  paths  in  session  design 

The  configuration  as  outlined  in  Figure  3  requires  two  types  of  communication:  uni-directional  and 
bi-directional. 

1.  Uni-directional  communication.  In  this  form  of  communication,  information  is  sent  fro  m  the 
control  room  to  the  pod  rooms.  Information  injects  and  tasks  can  be  read  and  evaluated  by  the 
pod  members,  but  cannot  be  changed.  The  control  room  will  not  provide  clarification  or  further 
information  based  upon  pod  requests.  T  his  approach  will  mimic  the  traditional  uni-directional 
communication  methods  often  used  i  n  complex,  extreme  events,  such  as  television,  r  adio 
reports,  phone  messages,  fax  and  memos.  Uni-directional  communication  will  also  be 
employed  by  the  control  r  oom  for  the  collection  of  data  fro  m  pod  rooms  and  individual  pod 
members.  Video,  audio  and  written  data  will  be  collected  for  exam  ination.  These  differ  ent 
methods  of  observation  will  provide  da  ta  for  the  analysis  of  the  collaborative  effort  and  the 
problem  solving  processes  and  outcomes,  as  well  as  provide  a  time  stamp  with  which  to  mark 
and  compare  all  actions  that  take  place  in  and  between  the  pods. 
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2.  Bi-directional  communication.  Information  will  be  shared,  discussed  and  exchanged  within  the 
pods  (herein  referred  to  as  intra-pod)  using  traditional  information  sharing  techniques.  The  se 
methods  include  verbal  and  written  communication  between  pod  members.  Information  may 
also  need  to  be  shared  between  some  or  all  of  the  pods  (inter-pod). 

For  inter-pod  communication  alternative  methods  of  collaboration  need  to  be  em  ployed. 
Subjects  may  need  or  want  to  communicate  with  one  another  inter-pod.  Indeed,  that  is  the  hope 
and  the  ai  m  of  the  task  w  ith  which  they  are  being  presented  during  two  of  the  four  overal  1 
periods  within  each  ses  sion.  They  will  need  to  hear  what  others  are  thinking  and  see  what 
problem  definition/problem  solving  they  are  pr  oposing.  During  two  periods  of  each  ses  sion, 
pod  members  will  need  to  be  able  to  have  input  or  at  least  understanding  and  consensus  with 
the  decisions  being  made  in  the  other  pods.  To  this  end,  each  pod  member  will  have  their  own 
computer  and  be  able  to  join  a  group  meeti  ng  or  start  new  private  meetings  with  oth  er 
participants. 

To  efficiently  achieve  both  uni- and  bi-directional  communication,  online  conferencing  software 
will  be  employed.  Using  online  conferencing  software  will  allow  for  information  to  be  passed  from 
the  control  r  ooms  to  the  pod  rooms  in  a  uni-directional  manner  and  collect  data  fro  m  each  pod 
room,  while  simultaneously  allowing  the  pods  to  communicate  with  one  another. 

With  the  online  conferencing  software,  the  control  room  will  be  able  to  transmit  injects  and  tasks  to 
the  pod  rooms  through: 

1.  Video  in  the  form  of  prefabricated  ‘news  reports’,  directly  through  video  conferencing  or  even 
through  video  emails  sent  to  the  pod  rooms; 

2.  Audio  radio  broadcasts,  phone  messages  and  other  traditional  forms  of  verbal  communication; 
and, 

3.  Text  presentation  simulating  fax  and  reporting  as  well  as  email  and  real  time  chat  conferencing. 

Conferencing  software  will  also  serve  as  a  condu  it  for  inter-pod  communication.  Subjects  will  be 
given  a  simple  video  tu  torial  explaining  the  basic  features  of  the  conferencing  software. 
Participants  will  be  able  to  use  video  chat,  audio,  instant  message  and  email  conference  features. 

In  order  to  ensure  that  both  quantitative  and  qualita  tive  information  is  collected,  the  control  room 
will  record  and  collect  data  using  both  the  c  onferencing  software  and  standard  recording 
equipment.  Online  conferencing  information  will  be  collected  through  the  conferencing  software. 
Each  meeting  will  be  recorded  in  a  screen  capture  video  and  notes  and  written  conversations  will  be 
collected  from  any  online  meetings.  The  recorded  digital  m  inutes  of  the  meeting  will  inc  lude  all 
video  and  audio  from  any  online  meetings  as  well  as  all  chat  and  em  ail  messaging.  An  additional 
benefit  to  t  his  method  is  that  all  decisions  made  in  the  conference  environm  ent  will  be 
automatically  timed  stamped.  This  will  help  to  understand  the  nature  of  the  decision  making  and 
collaborative  behaviours  employed  by  the  various  pods  in  all  the  different  runs  of  the  experiment. 
As  a  redundant  measure  to  further  guarantee  the  safe  collection  of  data  and  to  capture  the  intra-pod 
actions  conducted  outside  of  the  web  c  onference,  a  video  camera  will  be  film  ing  from  each  pod- 
room  ceiling  to  audio  and  video  record  events  in  each  pod  room. 

The  pods  may  need  or  want  to  perform  research  in  to  the  problems  they  are  trying  to  solve.  As  a 
result,  each  machine  will  be  equipped  with  internet  access,  and  be  loaded  with  Microsoft  Office  to 
perform  as  a  traditional  work  station  in  addition  t  o  serving  as  a  conferencing  point.  A  printer  will 
be  available  in  each  pod  room  and  will  be  linked  to  each  computer. 
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A  virtual  briefing/ debriefing  room  will  be  used  at  the  beginning  and  end  of  each  session.  This 
virtual  room  will  be  an  online  conference  that  w  ill  be  configured  identically  across  sessions.  This 
meeting  will  serve  three  functions: 

1.  It  will  introduce  the  study  process  and  ‘rules  of  engagement’  to  the  pod  part  icipants.  (Note: 
since  it  will  be  difficult  if  not  im  possible  to  gather  all  the  required  participants  in  one 
geographical  centre,  the  pod  rooms  will  be  set  up  in  different  locations,  as  required.) 

2.  It  will  provide  a  basic  video  tutorial  on  using  the  online  conferencing  software  and  answer  any 
questions  pertaining  to  the  software  and  its  proper  execution. 

3.  After  the  completion  of  the  experiment,  participants  will  rejoin  the  virtual  meeting  room  for  the 
purposes  of  debriefing. 

4.2  Control  Variable 

In  order  to  have  a  rigorous  experiment  design,  it  is  i  mportant  to  control  as  many  potentially 
influential  variables  as  possible.  There  are  various  control  variables  in  the  current  design  such  a  s 
controlling  for  subjects’  levels  of  authority  in  their  organizations,  pod  size,  introduction/briefing 
statements,  length  of  session,  etc.  While  many  of  these  have  been  identified  and  their  control  has 
been  built  into  the  design,  the  contro  1  variable  of  greatest  i  mportance  is  situation  complexity. 
According  to  the  Model  for  Inter-Organizational  Problem  Solving  developed  in  Task  1,  situation 
complexity  modified  by  assets  and  time  results  in  different  approaches  to  problem  solving.  Taking 
into  consideration  the  objectives  for  th  e  current  study  as  well  as  the  time  and  resources  allocated, 
the  decision  was  made  to  focus  solely  on  com  plex  situations,  therefore  controlling  situation 
complexity  at  the  complex  level.  Depending  on  this  study’s  results,  future  work  could  include 
converting  situation  complexity  from  a  control  variable  to  an  independent  variable  to  determine  the 
impact  that  different  levels  of  com  plexity  have  on  the  problem  solving  process.  The  current  stud  y 
will  result  in  tested  tools  to  assess  situ  ation  complexity.  These  tools  could  then  easily  be  used  in 
future  work  to  develop/assess  scenarios  according  to  various  levels  of  complexity. 

4.2.1  Control  variable:  Situation  complexity 

The  experiment  is  designed  to  m  easure  the  extent  to  which  1)  approach  to  multi-organizational 
decision-making;  and  2)  types  of  m  ulti-organizational  environments  impact  on  pr  oblem  solving 
processes  and  outcomes  during  complex  major  events.  As  a  result,  the  study  design  must  be  able  to 
hold  constant  the  control  variable  of  situation  complexity’,  while  manipulating  the  approaches  used 
to  problem  solve,  and  the  types  of  multi-organizational  environments.  Considerable  effort  will  be 
made  to  ensure  that  situation  com  plexity  is  held  constant  at  the  com  plex  level  within  and  across 
sessions. 

Situation  complexity  will  be  held  con  stant  by  employing  within  each  session  the  sa  me  in  vivo 
scenario  carefully  developed  and  thoroughly  assessed  as  representing  a  complex,  extreme  situation. 
The  scenario  will  be  presented  to  the  pods  and  evolve  throughout  the  testing  session  through  a 
series  of  injects  delivered  from  the  control  room  to  each  pod.  The  overall  process  will  be  similar  to 
that  of  a  functional  exercise,  a  method  frequently  used  for  training  and  planning  for  extreme  events 
(FEMA,  2003). 
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4.2. 1.1  Description  of  scenario  development  process 

As  per  best  practice  guidelines  for  deve  lopment  of  similar  types  of  exercises  ( FEMA,  2003),  the 
study  will  require  an  overall  com  plex  master  scenario  of  events,  accompanied  by  highly  detailed 
messages  and  injects  to  e  nsure  that  the  in  vivo  situation  remains  complex  throughout  the  testing 
session.  To  ensure  that  the  scenario  is  realistic,  the  development  process  will  consider  the  authority 
required  for  making  decisions,  experience  levels  of  participants,  resources  available  for  the  various 
organizations  involved,  and  the  ti  me  period  to  be  si  mulated,  including  m  essage  flow  and  the 
passage  of  time,  also  known  as  skip  time.  In  section  6.0  Implementation:  Next  Steps,  references  are 
made  to  the  specific  steps  required  to  develop  these  various  components. 

The  in  vivo  aspect  of  the  study  will  require  that  the  scenario  has  a  believable  narrative  to  guide 
participants  through  an  extreme  event,  compressing  the  simulation  into  approximately  four  hours. 
This  four  hour  block  would  include  the  brief  training  on  conferencing  software,  briefing/debriefmg 
sessions  and  transitions.  In  addition,  to  contribute  to  the  in  vivo  simulation  aspect  of  the  study  , 
scenario-based  messages  and  injects  will  be  delivered  via  several  communication  routes  including: 
video  news  flashes,  text  messages,  etc. 

In  order  to  e  nsure  that  the  measurable  level  of  situation  complexity  is  at  the  complex  level,  the 
research  team  will  appl  y  the  criteria  of  the  Model  for  Inter-Organizational  Problem  Solving 

outlined  in  T  ask  1  of  the  project  (Lem  yre  et  al.,  2009).  The  Model  outlines  three  key  factors 
contributing  to  situation  complexity:  (a)  Event  Impact,  (b)  Uncertainty,  and  (c)  Vulnerability. 
Contributing  to  each  of  these  factors  are  various  elements: 

•  Factor  A:  Event  Impact  is  shaped  by  the  elements  of  impact  scope,  impact  severity  and 
impact  timing,  as  well  as  media  involvement  and  political  processes. 

•  Factor  B:  Uncertainty  is  shaped  b  y  the  elements  of  novelty  of  the  situation,  knowledge 
about  the  hazard,  degree  of  pr  ior  planning,  availability  of  information,  fl  exibility  in 
interpreting  the  situation  using  m  ultiple  perspectives,  presence  o  f  new  organizations  and 
the  speed  at  which  the  context  changes. 

•  Factor  C:  Vulnerability  is  shaped  by  the  ele  ments  of:  levels  and  diversity  of  social 
characteristics  of  the  pop  ulations  involved,  of  economic  standing,  existence  of  social 
capital  and  community  competence,  type  and  quality  of  infor  mation  sources  and  of 
communication  networks,  and  functional  characteristics  on  the  critical  infrastructure. 

To  structure  the  scenario  and  ensure  that  all  key  factors,  elements  and  potential  sub-ele  ments  are 
considered  in  developm  ent,  the  team  has  deve  loped  a  grid  t  hat  provides  the  ke  y  elements 
contributing  to  each  of  these  factors  (  see  Figure  4).  By  following  this  grid  in  a  sy  stematic  manner, 
and,  where  appropriate,  e  mphasizing  the  complex  end  of  dimensions,  the  team  will  develop  and 
assess  scenarios  to  ensure  that  an  appropriate  level  of  com  plexity  is  achieved  and  m  aintained 
through  the  session.  ( Refer  to  Figure  4  for  Template  scenario  elements  grid). 
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Figure  4:  Template  scenario  elements  grid 


Experimental  Platform 

To  provide  a  context  or  platform  for  the  complex  event  scenario,  the  research  team  will  use 
Gapville  (see  Figure  5  below),  a  training  tool  created  as  part  of  the  Psychosocial  Risk  Manager 
(PRiMer)  training  program,  funded  by  the  Centre  for  Security  Sciences,  CRTI-06-0259TD 
project.  Created  for  use  in  learning  activities  and  tabletop  exercises,  this  cityscape  illustration 
contains  all  the  built  and  social  infrastructure  of  a  mid-sized  North  American  city.  Using  Gapville 
as  a  base,  and  applying  the  grid  of  complexity  factors,  elements  and  sub-elements,  the  research 
team  will  design  various  scenarios  that  can  be  assessed  to  determine  which  are  most  appropriate 
for  the  study. 

If  found  useful  to  build  comparisons  or  to  initiate  convergence  of  common  tools,  the  initial  basic 
scenario  storyboard  may  be  chosen  from  DRDC  current  bank  of  CBRNE  and  c  yber  threats  or 
collaborators’  sets  such  as  from  DHS,  FEMA,  EU  or  UK.  Then  features  will  be  added  to  match 
our  criteria  for  complexity  and  timeframe. 
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Figure  5:  Gapville 


4.2.1.2  Assessment  of  scenario  complexity  and  appropriateness 

The  research  team  will  initiall  y  develop  three  scenarios  and  related  injects  and  support  ing 
materials  in  order  to  select  the  most  appropriate  for  use  in  the  study  (only  one  scenario  will  be 
used  in  the  s  tudy  at  this  point).  Curre  ntly  we  are  contemplating  using  scenarios  relat  ed  to  a 
radiation-nuclear  attack,  a  cyber-attack,  or  a  natural  disaster.  Considerable  effort  will  be  expended 
in  developing  assessment  tools  to  measure  the  following: 

•  Complexity  assessments  for  each  of  the  four  periods  within  each  of  the  t  hree  draft 

scenarios.  The  assessment  tools  will  be  based  on  the  factors,  elements  and  sub  elements 
outlined  in  Figure  4  -  Template  scenario  elements  grid', 

•  Ratings  of  likelihood  of  occurrence  of  scenario; 

•  Level  of  appropriateness  given  subjects  experience  and  knowledge;  and 

•  Consistency  of  interpretation  of  key  facts  by  pod  members. 

Complexity  assessments 

Once  the  seri  es  of  three  draft  scenarios  has  been  developed,  the  research  team  will  assess  the 
complexity  of  each  of  the  four  periods  for  each  of  the  three  scenarios.  The  ratings  of  complexity 
will  be  according  to  a  standard  tool  developed  expressly  for  this  purpose  and  will  be  based  on  the 
the  factors,  elements  and  sub-elements  outlined  in  the  grid.  An  initial  draft  seal  ing  for  each  sub- 
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element  has  been  provided  in  the  Annex  C.  This  will  be  further  developed  once  the  series  of  three 
draft  scenarios  have  been  finalized. 

Using  the  sc  enario  complexity  assessment  tool,  eight  raters  4  will  be  briefed  on  the  elements  of 
situation  complexity  as  per  the  Model  outlined  in  Task  1.  Each  ra  ter  will  then  consider  each 
situational  sub-element  within  each  of  the  four  periods  within  each  of  the  three  s  cenarios 
developed  to  rate  the  overall  scenario  with  respect  to  complexity.  Judgements  from  each  rater  will 
be  examined  and  used  for  two  purposes.  The  initial  analysis  of  ratings  will  be  used  to  select  the 
most  appropriate  scenario  to  be  used  in  the  e  xperiment.  The  second  use  of  t  he  ratings  will  be  to 
improve  the  sub-ele  ments  of  the  selec  ted  scenario.  This  would  be  achieved  by  enhancing  or 
decreasing  certain  sub-elements  of  the  chosen  scenario  based  on  raters’  feedback. 

Assessment  of  likelihood  of  occurrence 

Given  the  in  vivo  aspect  of  the  study,  it  will  be  important  to  select  a  scenario  that  has  a  potential 
to  actually  occur.  As  a  result,  the  resear  eh  team  will  also  develop  a  second  brief  tool  t  o  have  the 
raters  performing  the  complexity  assessments  also  provide  ratings  and  assessments  of  the  realistic 
nature  of  the  seen  ario,  injects  and  materials.  They  will  be  asked  to  provide  quantitative  and 
qualitative  ratings  of  both  the  extent  to  which  the  overall  scenario  would  be  likely  to  occur,  and  to 
identify  any  aspects  of  the  scenario  or  materials  that  were  unlikely  or  not  realistic. 

Assessment  of  level  of  appropriateness  of  scenarios 

The  raters  performing  the  complexity  assessment  will  also  be  asked  to  comment  upon  and  assess 
the  extent  to  which  the  level  of  problem  solving  and  scenario  elements  would  be  appropriate  for 
the  likely  study  subjects.  The  raters  will  be  provided  with  a  brief  description  of  the  intended  study 
subjects  with  respect  to  levels,  backgrounds  and  exp  erience,  and  a  tool  to  capture  their  ratings  of 
appropriateness  for  various  aspects  of  the  scenarios,  injects  and  materials. 

Consistency  of  interpretation  of  key  facts  by  subjects 

It  will  be  i  mportant  that  the  selected  scenario  s,  injects  and  material  s  are  understood  and 
interpreted  consistently  across  subject  s.  The  assessment  of  interpretati  on  consistency  will  be 
undertaken  only  once  the  raters  have  completed  their  various  assessments  and  one  scenario  has 
been  selected.  This  assessment  will  take  place  during  the  pilot-testing  of  the  overall  study 
sessions  by  having  pilot-test  subject  s  review  and  provide  clarity  ratings  on  all  injects  an  d 
materials,  and  to  ask  for  a  sample  of  interpretations  across  pilot-test  participants.  An  assessment 
tool  to  elicit  and  capture  this  information  will  be  developed  for  the  pilot-testing  stage. 


4.3  Independent  Variables 

4.3.1  Independent  variable:  Type  of  multi-organizational  decision-making 
approach 

A  primary  independent  variable  for  the  study  will  be  multi-organizational  decision-making 
approach.  As  hypothesized  in  the  Model  described  in  Task  1,  problem  solving  in  multi- 
organizational  context  should  atte  mpt  to  reduce  situation  complexi  ty  in  or  der  to  effectivel  y 
implement  solutions.  The  extent  to  which  a  specific  type  of  approach  is  optimal  depends  on  the 


4  Raters  will  be  a  mix  of  experienced  responders,  decision-makers  or  trainers  of  various  backgrounds. 
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actual  level  of  situation  com  plexity  encountered,  and  can  be  modified  by  the  assets  available  to 
each  organization,  and  the  various  sta  ges  of  probl  em  solving.  As  described  i  n  the  overview  in 
Section  2. 0,  the  goal  of  the  present  study  is  to  effe  ctively  manipulate  both  multi-organizational 
approach  and  m  ulti-organization  context  to  test  the  optimal  approach  to  problem  solving  in 
complex  situations,  as  measured  through  decision  outcomes  and  processes. 

The  multi-organizational  approach  will  be  manipulated  during  the  experi  ment  by  releasing  four 
types  of  task  instructions  to  the  pods.  Four  of  the  eight  sessions  will  be  focused  on  coordinating 
tasks,  while  the  other  four  sessions  will  be  focused  on  collaborating  tasks.  The  tasks  will  take  the 
form  of  specific  instructions  for  the  pod  mem  bers  to  develop  a  product  (e.g.,  definitions  of  the 
main  problem,  prioritised  list  of  options)  using  a  specific  approach  within  the  six  problem  solving 
stages  (i.e.,  problem  identification,  problem  definition,  solutio  n  generation,  decision  making, 
solution  implementation,  and  feedback  on  success/failure). 

The  main  design  principles  of  the  tasks  include: 

•  Tasks  will  consist  of  instructions  for  a  group  task  that  outline:  a)  the  approach  that  should  be 
followed  (e.g.,  work  inde  pendently,  consult  each  pod  m  ember,  consult  other  pods,  gain 
complete  consensus);  and  b)  the  product  that  needs  to  be  develo  ped  (e.g.,  table  outlining 
most  likely  impacts  of  potential  actions). 

•  Tasks  will  include  time  limits  for  the  task  completion  (e.g.,  30  minutes). 

•  Tasks  will  be  concise  (less  than  1  page)  and  easily  interpretable  -  limited  time  during  the 
session  should  be  used  by  pod  members  trying  to  clarify  what  is  required.  Most  of  the  time 
should  be  used  to  work  on  the  tasks. 

•  Tasks  will  coincide  with  events  happening  in  the  scenario. 

•  The  various  tasks  will  cover  three  stages  of  problem  solving,  i.e.,  problem  definition, 
solution  generation  and  decision  making. 

•  During  a  session,  identical  tasks  will  be  administered  to  each  pod  at  the  same  time  fro  m  the 
control  room  via  the  online  conferencing  system. 

•  Tasks  will  be  developed  in  such  a  manner  as  to  ensure  that  the  same  problem  solving  stages 
are  occurring  in  each  group  (i.e.,  problem  definition,  solution  gene  ration,  decision  making), 
and  the  same  approach  to  problem  solving  is  being  used  (i.e.,  coordination  or  collaboration). 

•  The  number,  timing  and  stage  of  proble  m  solving  of  tasks  will  be  identical  across  sessions, 
regardless  of  session  type. 

4.3. 1.1  Description  of  task  development  process 

In  order  to  develop  tasks  that  effectively  manipulate  the  inter-organizational  approach  to  problem 
solving,  the  tasks  will  be  developed  based  on  a  3 -way  grid  outlining  the  analysis  of  the  four  inter- 
organizational  approaches  by  levels  o  f  shared  assets  according  to  the  six  stages  of  proble  m 
solving  (see  Annex  D).  By  having  operationalized  the  six  stages  of  problem  solving  into 

observable  behaviours  as  outlined  in  the  grid,  the  research  team  will  be  able  to  design  task 
directions  that  will  elicit  the  specific  behavi  ours  associated  with  either  coordinating  or 
collaborating  at  the  problem  definition,  solution  generation  and  decision  making  stages.  Based  on 
the  3-way  grid  analysis,  the  task  development  process  will  follow  the  design  principles  outlined 
above.  While  four  different  tasks  will  be  require  d  to  im  plement  the  study  ,  initially,  we  will 
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develop  twelve  tasks  (4  X  3)  fro  m  which  the  most  appropriate  four  will  be  chosen,  o  nee 
calibration  has  occurred. 

4.3.1.2  Assessment  process  for  problem  solving  tasks 

The  problem  solving  tasks  used  in  the  scenario  will  undergo  a  similar  assessment  process  to  that 
used  for  rating  scenario  com  plexity.  This  will  ensure  that  the  task  s  will  be  able  to  elicit  either 
coordination  or  collaboration  behaviour  during  each  of  four  periods  during  the  study  session.  The 
same  raters  who  will  be  undertaking  the  complexity  assessment  will  be  initi  ally  asked  to  judge 
the  extent  to  which  each  o  f  the  tasks  are  likely  to  elicit  collaborative  or  coordinating  behaviours 
based  on  the  key  definitions  and  behaviours  outlined  in  the  original  analy  tic  3 -way  grid.  An 
assessment  tool  designed  to  elicit  and  collect  these  ratings  will  be  developed  based  on  the  analytic 
3 -way  grid.  Ratings  will  be  used  to  not  only  select  the  most  appropriate  tasks,  but  also  to  improve 
the  set  of  tasks  ultimately  selected  for  the  study. 

In  addition  t  o  the  ratings  provided,  the  selected  tasks  will  be  pilot-tested  with  sm  all  groups  of 
three  people  to  observe  the  extent  to  which  collaborative  and  coordinating  behaviours  are  actually 
elicited  by  the  various  tasks.  Tasks  will  be  ad  justed  accordingly  to  ensure  that  the  y  are 
appropriate  for  eliciting  the  desired  coordination  or  collaborating  behaviours.  For  this  process  , 
observation  tools  will  be  developed  that  op  erationalize  coordinating,  cooperating  and 
collaborating  behaviours  to  the  extent  that  the  y  can  be  monitored,  observed,  recorded  and 
assessed.  These  tools  will  be  further  refined  during  the  pilot-testing  phase  to  be  used  for  coding 
and  observing  all  8  sessions.  This  will  provide  an  internal  validity  check  to  the  study  to  confirm 
that  both  coordination  and  collaboration  behaviours  were  elicited  from  the  tasks  during  the  in  vivo 
study. 

4.3.2  Independent  variable:  Multi-organization  environment 

The  second  independent  variable  that  will  be  included  in  the  study  is  multi-organization 
environment.  As  described  in  Task  1,  complex  problems  usually  require  multiple  organizations  to 
work  together.  This  sectio  n  describes  how  different  types  of  organizations  will  be  grouped  into 
pods  to  simulate  multi-organizational  environments.  Section  4.3.2. 1  Intra  pod  description 
describes  the  variations  within  pod  com  positions,  while  section  4. 3. 2. 2  Inter  pod  description 
presents  the  variations  in  interactions  between  pods  within  each  session  .  Section  4. 3. 2. 3 
Description  of  session  and  pod  composition  describes  the  ways  that  these  pods  will  be  organized 
according  to  session  design. 

As  previously  outlined,  during  each  session,  thr  ee  pods  of  three  individuals  each  will  receive 
scenario  injects  and  instructions  from  a  single  control  room.  Each  pod  wi  11  be  located  in  a 
separate  room  but  will  have  the  opportunity  to  communicate  with  the  other  pods  at  certain  stages 
during  the  session.  The  control  room  will  observe  behaviours  both  within  the  pods  and  between 
the  pods.  See  Figure  6  below  for  a  depiction  of  a  session  design  set  up. 
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Figure  6:  Overall  session  composition 


4.3.2. 1  Intra  pod  description 

Pods  will  be  made  up  of  three  organization  t  ypes:  Military  (M),  ICS  non-military  (I),  and  Non- 
ICS  (NI).  During  the  experi  ment,  the  control  r  oom  will  observe  problem  solving  processes  and 
outcomes  within  the  pods.  Following  the  completion  of  the  experim  ent,  further  analy  sis  of  the 
data  collected  from  the  pods  will  illuminate  diffe  rences  and  similarities  in  the  ways  that  people 
collaborated  depending  on  the  co  mposition  of  the  grou  p  and  depen  ding  on  whether 
communication  was  open  or  closed.  It  therefor  e  becomes  necessary  to  discuss  the  different 
possibilities  for  pod  com  position.  There  are  tw  o  possibilities  for  pod  com  position  within  t  he 
experimental  design,  which  are  outlined  below. 


Table  3:  Pod  composition  descriptions 


Pod  composition 

Description 

Example 

Homogeneous  Pod 

(Same  organization) 

Three  members  from  the  same 
organizational  type  (Military  (M), 
ICS  non-military  (I),  or  Non-ICS 
(NI)) 

A  military  pod  in  which  all 
three  participants  are  from  the 
military;  or  all  from  three  social 
service  agencies 

Mixed  Organizations  Pod 

Three  members  each  from  the 
three  different  organizational 
types  (Military  (M),  ICS  non¬ 
military  (I),  and  Non-ICS  (NI)) 

A  mixed  organization  pod  made 
up  of  a  participant  from  the 
military,  a  participant  from  a 
police  department,  and  a 
participant  from  a  non¬ 
governmental  organization 

4.3.2.2  Inter  pod  description 

In  addition  to  m  onitoring  and  recording  comm  unication,  information-seeking,  decision  making 
and  collaborative  behaviour  within  the  pods,  the  experiment  will  also  record  these  behaviours  as 
they  occur  between  the  pods.  Later  analysis  will  look  at  the  differences  in  behaviour  according  to 
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the  overall  composition  and  intera  ction  during  each  session.  There  are  four  possibl 
configurations  for  each  session  of  the  experiment;  these  configurations  are  described  below. 


e 


Table  4:  Configuration  descriptions 


Configuration 

Pod  composition 

Inter-pod 

interaction 

Description 

A 

Homogeneous  Pod 

No 

communication 

No  communication  between  the  three 
pods,  each  with  all  three  members  from 
the  same  type  of  organizational 
stmcture: 

•  Military  (M)  pod 

•  ICS  non-military  (I)  pod 

•  Non-ICS  (NI))  pod 

B 

Homogeneous  Pod 

Open 

communication 

(  M  M  M  J 

f  iii  w - ►/nininm 

Open  communication  between  the  three 
pods,  each  with  all  three  members  from 
the  same  type  of  organizational 
stmcture: 

•  Military  (M)  pod 

•  ICS  non-military  (I)  pod 

•  Non-ICS  (NI)  pod 

C 

Mixed 

Organizations  Pod 

No 

communication 

No  communication  between  the  three 
mixed  organizations  pods,  each  with  one 
member  from  each  of  the  types  of 
organizational  structures: 
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Configuration 

Pod  composition 

Inter-pod 

interaction 

Description 

•  3  X  Mixed  organizations  pod 

(Military  (M),  ICS  non-military 
(I),  and  Non-ICS  (NI)) 

D 

Mixed 

Organizations  Pod 

Open 

communication 

M  Nl  M  j 

M  NI  M  1* - ►(  1  Nl  M  J 

Open  communication  between  the  three 
mixed  organizations  pods,  each  with  one 
member  from  each  of  the  types  of 
organizational  structures: 

•  3  X  Mixed  organizations  pod 

(Military  (M),  ICS  non-military 
(I),  and  Non-ICS  (NI)) 

The  changing  make-up  of  the  pods  is  described  in  Table  5  below.  The  subscript  numbering  refers 
to  individual  participants  and  it  shows  how  these  individuals  are  rearranged  during  the  four 
configurations. 


Table  5:  Pod  members  by  configuration 


Pod  1 

Pod  2 

Pod  3 

Config. 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant 

A 

M, 

m2 

m3 

ii 

I2 

u 

NI, 

ni2 

ni3 

B 

M, 

m2 

m3 

ii 

h 

h 

NI, 

ni2 

ni3 

c 

M, 

h 

NI, 

m2 

h 

ni2 

m3 

I3 

ni3 

D 

M, 

ii 

NI, 

m2 

i2 

ni2 

m3 

I3 

ni3 

4.3.2.3  Description  of  session  and  pod  configuration 

There  are  two  broad  appr  oaches  for  the  session  design  in  which  all  four  configurations  will  be 
used  (A,  B,  C  and  D;  based  on  co  mposition  and  inter-pod  inte  raction).  These  approaches  are 
defined  by  coordination  and  collaboration  and  include  sessions  that  focus  on  coordinating  tasks  as 
well  as  sessions  that  focus  on  collaborating  tasks.  The  order  of  the  four  configurations  will  rotate. 
Differences  with  respect  to  collaborative  or  coordinating  behaviour  will  be  noted  according  to  the 
ordering  of  these  configurations.  See  T  able  6  be  low  for  the  arrange  ments  of  configurations 
according  to  problem  solving  approach. 
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Table  6:  Stage  ordering  by  pod  configuration  and  problem  solving  approach 


Session 

No. 

Stage  1 

Stage  2 

Stage  3 

Stage  4 

Approach 

1 

A 

B 

C 

D 

coordination 

2 

C 

D 

A 

B 

coordination 

3 

B 

A 

D 

C 

coordination 

4 

D 

C 

B 

A 

coordination 

5 

A 

B 

C 

D 

collaboration 

6 

C 

D 

A 

B 

collaboration 

7 

B 

A 

D 

C 

collaboration 

8 

D 

C 

B 

A 

collaboration 

The  number  of  participants  that  are  needed  for  the  experiment  are  described  in  Table  7  in  terms  of 
the  organizational  type  of  the  participants’  workplace. 

Table  7:  Number  of  participants  needed  according  to  organizational  type 


Organizational  Type 

No.  Participants  per  Session 

No.  Participants  For  Experimental  Design 

Military  (M) 

3 

24 

ICS  non-military  (I) 

3 

24 

Non-ICS 

3 

24 

All 

9 

72 

4.4  Dependent  variables 

As  outlined  in  Section  2.1,  the  dependent  variables  for  th  e  study  will  focus  on  the  pr  oblem 
solving  processes  and  outcomes.  Given  that  the  sessions  will  be  v  ideo  and  audio-recorded,  this 
will  provide  copious  amounts  of  rich  process  data  that  can  be  assessed  to  de  termine  the  extent  to 
which  independent  variables  are  having  an  impact  on  the  problem  solving  processes.  Similarly, 
the  tasks  developed  will  result  in  problem  solving  “products”  and  docum  entation  of  decisions. 
These  will  constitute  the  source  data  for  measurement  of  problem  solving  outcomes. 

Overall,  there  will  be  two  sets  of  dependent  variables:  one  set  focusing  on  problem  solving 
processes',  a  second  set  focusing  on  problem  solving  outcomes.  The  main  dependent  variables 
outlined  for  problem  solving  processes  include: 

•  Satisfaction  with  problem  solving  process  (individual,  collective,  external  panel); 

•  Level  of  participation  with  each  pod; 

•  Level  of  participation  between  pods; 

•  Time  spent  on  problem  solving  stages; 

•  Pattern  of  engagement  in  problem  solving  stages;  and 

•  Task  and  group  cohesion. 
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The  main  dependent  variables  outlined  for  problem  solving  outcomes  include: 

•  Decision  quality  (individual,  collective,  external  panel); 

•  Satisfaction  with  problem  solving  outcome  (individual,  collective,  external  panel); 

•  Level  of  agreement  on  outcome;  and 

•  Changes  in  individual  and  collective  goals. 

•  Time  to  reach  decision 

Various  measures  will  be  developed  for  each  of  these  variables.  Measures  will  take  the  forms  of 
self  reports,  participant  ratings,  observation,  and  time/content  coding.  More  detailed  information 
and  measurement  considerations  for  each  of  the  proposed  dependent  variables  are  contained  in 
Annex  E. 


4.5  Analysis  plan 

Once  the  dependent  variable  measures  have  been  finalized,  the  detailed  analysis  plan  will  be 
developed  for  the  experimental  component.  Initially,  once  data  have  been  coded  and  cleaned,  the 
team  will  u  ndertake  extensive  descriptive  analyses  for  each  m  easure  to  assess  basi  c  data 
characteristics  such  as  outliers,  normality  ,  linearity,  and  homoscedasticity.  Once  we  have 
determined  the  basic  data  characteristics,  the  primary  analyses  are  anticipated  to  include  analyses 
of  variance  and  covarianc  e  (ANOVA,  ANCOVA,  MANOVA,  MANCOVA).  This  will  result  in 
the  testing  o  f  the  main  effects  of  each  of  th  e  three  independent  variables,  along  with  the 
interactions  between  variables.  Contingent  on  the  results  of  the  analy  ses  of  variance  and 
covariance,  post-hoc  comparison  testing  will  be  u  ndertaken  to  further  inter  pret  main  effect  s 
and/or  interactions. 

Depending  on  the  final  sele  ction  of  dependent  variable  measures,  there  may  be  so  me 
opportunities  to  further  explore  the  findings  from  the  pri  mary  analyses  using  correlational 
techniques  to  understand  the  relationships  among  certain  dependent  variables.  This  will  facilitate 
a  better  overall  understanding  of  problem  solving  processes  and  outcomes.  Consideration  will  be 
given  to  integrating  these  types  of  analyses  in  the  detailed  analysis  plan. 
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5  Feasibility  and  Contingency  Analysis 


5.1  Feasibility  of  experimental  plan 

In  Section  2.2  Overview  of  resear  eh  strategy,  some  of  the  theoretical  re  search  challenges  and 
limitations  of  the  propose  d  approaches  to  the  experiment  are  identified.  Ackn  owledging  these 
conceptual  challenges,  this  section  focuses  on  identifying  the  m  ore  practical  issues  of 
implementing  a  study  of  the  scope  and  breadth  pr  oposed  herein.  The  more  significant  practical 
issues  centre  on: 

♦  Recruitment  of  the  requisite  number  of  study  participants 

♦  Identification  and  selection  of  the  most  appropriate  types  of  participants 
(organizational  level,  experience,  etc) 

♦  Participant  availability  (location  /tim  e)  for  study  session  and  handling  “no  shows” 
due  to  urgent  matters  to  be  attended  to 

♦  Technology  requirements  for  the  conduct  of  the  study 

♦  Ethics  approvals 

Failure  to  take  into  account  these  issues  over  the  timeline  of  the  experiment  has  the  potential  to 
negatively  impact  the  study  outcomes.  The  study  design  as  proposed  though,  is  feasible.  Although 
there  may  be  specific  elements  of  the  design  that  appear  daunting  to  im  plement  (e.g.,  number  of 
recruits  needed),  we  view  this  as  more  of  a  “complicated”  problem  rather  than  a  “wicked” 
problem.  The  experimental  design  employs  an  approach  similar  to  the  proven  Hydra  system  and 
technologies  for  data  tr  ansmission  and  capture  are  now  very  rob  ust  and  user  ffiendl  y. 
Anticipatory  planning  and  attention  to  detail  during  the  implementation  cycle  will  mitigate  many 
of  the  risks  associated  with  the  issues  noted  above. 


5.2  Risks  and  contingencies 

This  section  sets  out  the  r  isks  and  specific  mitigation  strategies  for  each  of  the  issues  identified 
above.  Note  that  these  risks  pertain  specifically  to  Component  2:  Experiment,  with  the  exception 
of  Risk  6:  Delays  in  obtaining  ethics  approvals,  which  appl  ies  equally  to  Component  1 : 
Qualitative  Interviews  and  Component  2  Experimental  design. 

Risk  1:  Unable  to  recruit  requisite  number  of  study  participants 

The  study  design  indicates  the  need  for  a  total  of  72  participants  for  Component  2:  Experiment. 
These  are  needed  in  increments  of  nine  for  8  sessions  over  the  implementation  timeline.  The  large 
number  of  p  articipants  requires  an  aggressive  recruitment  drive  of  potential  candidates.  While 
recruitment  of  participants  for  the  sim  ulations  is  according  to  contract  the  responsibilit  y  of  the 
Technical  Authority  (TA),  it  cannot  be  relied  on  exclusively  to  meet  the  needs  if  the  experiment  is 
to  be  a  success.  The  strategy  will  be  to  share  responsibilit  y  for  recruitment  between  the  TA  and 
the  University  of  Ottawa,  with  the  proj  ect  team  focusing  more  on  recruitment  efforts  outside  of 
the  military  and  RCMP.  There  will  be  a  need  to  start  the  recruitment  process  early  and  to  take  into 
account  a  30%  drop-out  rate  (refer  to  Risk  3). 

Risk  2:  Unable  to  select  most  appropriate  types  of  participants 
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In  addition  to  recruit  ment  of  a  large  n  umber  of  study  participants,  the  partici  pants  must  meet  a 
number  of  selection  criteria.  These  criteria  have  been  specified  in  Section  4.1.2  Description  of 
subjects/sample.  The  criteria  have  been  developed  to  include  as  broad  a  spectru  m  of  potential 
participants  as  possible  and  still  make  pod  compositions  realistic  and  credible  for  the  type  of 
scenario  presented.  The  relatively  senior  level  of  participants  will  make  their  recruitment  difficult, 
given  the  time  demands  of  their  jobs.  With  some  creative  scheduling  and  lim  iting  the  length  of 
the  simulation  to  half  a  day  in  total,  this  risk  will  be  somewhat  mitigated. 

Risk  3:  Scheduling  conflicts  affect  participant  availability  for  a  planned  session 

Experience  indicates  that  scheduling  conflicts  will  occur  and  participants  will  decline  attendance 
at  the  last  moment  due  to  urgent  matters/sickness,  putting  the  running  of  a  session  in  jeopardy.  To 
mitigate  this  risk,  for  each  session,  three  to  four  additional  participants  will  be  recruited  and 
placed  on  “call”.  These  individuals  will  be  utilized  on  an  as  needed  basis.  Two  make-up  sessions 
will  also  be  planned  towards  the  end  of  the  implementation  phase.  Individuals  on  “call”  who  have 
not  participated  up  to  this  point  will  have  an  opportunity  to  do  so.  In  addition  to  the  above ,  the 
timing  of  the  initial  8  sessions  will  take  into  account  the  periods  when  participants  are  most  likely 
to  be  available. 

Risk  4:  Hardware/software  utilized  fails  to  perform  as  per  specifications 

As  with  the  utilization  of  any  computer  hardware  and/or  software,  there  is  a  ri  sk  that  it  will  not 
work  when  you  need  it  and  the  simulation  process  will  be  disrupted.  These  risks  will  be  mitigated 
by  choosing  proven  equ  ipment  and  software,  training  technical  staff  in  advance  on  the 
equipment/process  and  providing  technical  staff  at  each  physical  location  to  troubleshoot 
problems,  if  indeed  they  should  arise.  A  technical  run  though  will  be  made  before  commencement 
of  each  session  to  verify  that  the  equipment  and  systems  are  operational. 

Risk  5:  Delays  in  obtaining  ethics  approvals  negatively  affect  project  schedule 

Obtaining  ethics  approva  1  for  a  research  design  of  this  type  is  an  onerous  task.  It  places  a 
significant  administrative  burden  on  project  resources.  Delays  in  obtaining  approvals  may  occur 
either  in  the  preparation  of  the  myriad  documents  or  in  the  actual  review  process.  Any  significant 
delay  will  impact  on  the  windows  of  opportunity  to  hold  study  sessions.  Fortunately  ,  the  project 
team  is  familiar  with  the  ethics  sub  mission  and  approval  process  and  the  Ethics  Review  Board  is 
sensitive  to  the  ti  me  demands  of  major  research  initiatives.  The  chief  mitigating  strategy  is  to 
prepare  and  subm  it  early  a  well  documented  st  udy  design  that  addresses  all  the  ethics 
requirements  and  avoid  requests  for  additional  clarifying  information.  If  clarifications  are  needed, 
the  focus  will  be  on  a  quick  turn-around  of  the  necessary  forms. 
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6  Implementation:  Next  Steps 


6.1  Component  1  -  Qualitative  interviews 

6.1.1  Prepare  interview  guide 

As  an  initial  step,  the  instructions  that  guide  the  administration  and  implementation  of  the 
qualitative  interviews  will  be  set  out  in  an  interview  guide.  The  guide  will  ensure  consistency 
across  interviews,  and  thus  increase  the  reliability  of  findings.  The  guide  will  include  what  to  say 
to  the  interviewee  before,  during  and  after  the  interview  and  what  to  do  during  and  following  the 
interview.  Questions  included  in  the  guide  will  focus  on  the  concepts  of  complexity  of  the 
situation,  multi-organizational  problem  solving,  and  the  multi- organizational  environment.  The 
exact  wording  of  the  questions  may  vary  slightly  depending  on  the  case  study  under  examination. 
Given  the  length  of  the  interview  (i.e.,  1  hour),  approximately  10  main  questions  will  be  used 
along  with  additional  probes  (see  Annex  B).  The  guide  will  include  a  standard  informed  consent 
form. 

6.1.2  Prepare  ethics  submission  and  obtain  approval 

The  qualitative  interview  co  mponent  of  the  study  will  requi  re  ethics  approval  as  per  the 
University  of  Ottawa  guidelines  and  the  Tri-  Council  Policy  Statement:  Ethical  Conduct  for 
Research  Involving  Humans  (TCPS).  Upon  accept  ance  of  Task  2  by  the  Technical  Authorit  y, 
separate  ethics  sub  missions  will  be  undertaken  for  Component  1  and  Component  2  are  as  of 
research.  This  will  ensure  a  faster  turn-around  time  for  approval  related  to  Component  1 . 

6.1.3  Identify/select  interviewees 

The  case  studies  documented  in  Task  1  will  serve  as  a  starting  point  for  identifying  and  selecting 
interviewees.  Three  to  four  case  stu  dies  will  be  utilized  to  co  mpile  a  li  sting  of  potential 
interviewees.  Identification  and  select  ion  of  po  tential  interviewees  will  be  done  with  the 
assistance  of  DRDC.  The  aim  will  be  t  o  maximize  the  number  of  interviews  at  one  location  for 
increased  efficiency. 

6.1.4  Test  interview  process/instruments,  train  interviewers 

Both  interview  process  and  instruments  will  be  pilot  tested.  The  questions  posed  in  the  guide  will 
be  vetted  by  a  small  expert  panel  for  relevancy  and  consistency  with  the  stated  research  questions. 
Two  interviewers  will  be  given  a  thorough  review  of  the  SDM  If  amework  and  the  rationale  for 
question  choices.  They  will  have  several  oppor  trinities  to  practice  the  in  terview  process, 
particularly  the  use  of  probing  questions.  Based  on  the  pilot  tests,  adjustments  will  be  made  to  the 
interview  questions  and/or  process  as  appropriate. 

6.1.5  Conduct  interviews 

The  project  schedule  indicates  that  the  qualitative  interviews  will  start  late  su  mmer  2010,  and 
continue  over  the  Fall  period.  Interviewee  availability  will  determine  the  exact  start  and  end  dates 
of  this  step.  Completion  of  interviews  by  late  Fall,  2010  provides  adequate  time  for  collation  and 
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analysis  of  the  data  and  the  preparation  of  an  interim  summary  on  findings  by  mid-January,  2011. 
This  will  permit  incorporating  the  findings  into  the  interpretation  of  the  in  vivo  simulation  results. 


6.2  Component  2  -  In  Vivo  Simulation  Experiment 

6.2.1  Development  and  calibration  of  scenarios 

A  number  of  scenarios  (3)  will  be  developed  utilizi  ng  the  grid  of  complexity  factors,  elements 
and  sub-elements.  Next  steps  will  be  to  fi  nalize  these  scenarios  and  assess  which  is  most 
appropriate  for  the  study .  Several  experts  will  be  asked  to  review  each  of  the  scenarios  for  f  ace 
validity.  After  adjustments  are  made,  raters  will  be  recruited  from  within  the  University  of  Ottawa 
to  rate  each  of  the  scenarios  in  ter  ms  of  their  level  of  complexity.  Each  potential  rater  will  attend 
a  briefing  on  the  SDM  fram  ework  and  the  definitio  ns  of  situational  co  mplexity.  To  ensure  the 
potential  raters  are  suitable,  a  multiple  choice  test  will  be  adm  inistered  after  the  briefing,  to 
determine  if  they  retained  the  necessary  information  to  prove  a  reliable  rater.  On  ce  accepted,  the 
raters  will  be  presented  with  the  master  scenario  of  events,  and  the  set  of  scales  designed  to  rate 
each  element’s  situation  complexity.  Raters  will  be  asked  to  fill  out  all  the  scales.  The  goal  is  to 
achieve  inter-rater  reliability  among  raters  as  to  what  constitutes  a  “complicated”  situation  versus 
a  “complex”  situation.  Based  on  rater  feedback,  th  e  most  appropriate  scenario  will  be  chosen  for 
the  study  and  modifications  made,  if  required,  before  launching  the  experiment. 

6.2.2  Development  and  calibration  of  problem  solving 

As  indicated  earlier,  the  problem  solving  tasks  used  in  the  scenario  will  be  submitted  to  a  simil  ar 
calibration  process  to  that  used  for  r  ating  scenario  complexity.  Once  the  most  appropriate 
scenario  has  been  selected  ,  eight  potential  tasks  will  be  reviewed  by  a  number  of  independent 
raters  (5)  to  j  udge  the  extent  that  each  will  elicit  collaborative  or  coordinating  behaviours  based 
on  the  key  definitions  and  behaviours  described  in  the  analytic  3 -way  grid.  An  instrument  will  be 
prepared  to  document  the  rater  judgements.  As  a  check  on  these  judgements,  the  tasks  will  also  be 
pre-tested  in  several  s  mall  group  settings.  The  f  our  tasks  that  consistently  receive  the  highest 
ratings  from  both  approaches  to  calibration  will  be  included  in  the  study. 

6.2.3  Finalization  of  measures  for  problem  solving  process  and  outcomes 

As  indicated  in  Annex  E,  a  number  of  measures  will  need  to  be  developed  in  house.  Additionally, 
measures  will  be  adapted  from  existing  measures  on  task  and  group  cohesion .  Corresponding 
forms  will  need  to  be  developed  for  both  the  measures  developed  in-  house  and  the  adapted 
measures.  The  self  report  measures  and  forms  will  be  tested  in  order  to  ensure  that  the  questions 
are  clear.  Processes  will  be  created  for  time  coding  video  and  audio  reco  rdings  and  a  codebook 
will  be  produced  for  content  coding. 

6.2.4  Develop  detailed  study  protocols 

The  effective  conduct  of  both  components  of  th  e  study  will  require  development  of  detailed 
research  protocols.  The  specific  require  ments  for  calibration  of  the  scenario  and  tasks  and  for 
finalizing  the  measures  for  problem  solving  processes  and  outcomes  have  already  been  identified. 
In  addition,  specific  protocols  will  need  to  be  developed  for 

•  recruiting  and  registering  study  participants; 
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•  obtaining  informed  consent; 

•  introducing  the  experiment  to  participants  (briefing  materials); 

•  sequencing  activities  /  tasks  within  the  simulation; 

•  using  the  dat  a  collection  instruments  both  qu  alitative  and  quantit  ative  to  track  problem 
solving  processes,  outcomes  and  participant  observations; 

•  facilitating  the  de-briefing  process; 

•  maintaining  participant  confidentiality;  and, 

•  ensuring  secure  storage  and  retention  of  completed  study  materials. 

6.2.5  Prepare  submission  and  obtain  ethics  approval 

Ethics  approval  is  required  before  i  mplementing  the  proposed  resear  eh  design  for  in  vivo 
simulation  of  meta-organizational  shared  decision  making.  Research  protocols  demand  that  “the 
University  of  Ottawa  and  its  investiga  tors  and  students  whose  resear  eh  projects  involve  the 
participation  of  human  beings  as  research  subj  ects  must  comply  with  the  Tr  i-Council  Policy 
Statement:  Ethical  Conduct  for  Research  Involvi  ng  Humans  (TCPS),  the  University  of  Ottawa’s 
guidelines,  and  other  relevant  guidelin  es  and  legislation.  This  is  both  a  collective  and  indiv  idual 
responsibility”  (Research  Ethics  Board  Project  Submission  Form,  2005,  page  13).  Thus,  a  key 
next  step  will  be  to  undertake  a  for  malized  ethics  approval  process  and  prepare  a  detailed 
submission  to  the  Univers  ity’s  Research  Ethics  Board  for  approval.  The  ethics  approval  process 
will  also  address  the  issue  of  participant  safety.  While  it  is  unlikely  that  participants  will  have  had 
direct  experience  with  the  exact  scenarios  that  will  be  developed,  there  is  a  remote  possibility  that 
the  scenario  may  trigger  a  high  stress  reaction.  Th  e  very  nature  of  the  scenari  o,  along  with  the 
injects,  visuals,  and  video  will  be  desig  ned  in  such  a  way  as  to  reduce  stress  b  y  avoiding  the  use 
of  imagery  or  subject  matter  with  a  high  dread  factor.  The  submission  documents  will  need  to 
summarize  the  research  protocols,  methodologies,  specifics  on  recruitment  and  selection  of 
participants,  risks  and  benefits  of  proposed  res  earch  (including  measures  to  minimize  or  mitigate 
risks),  the  informed  consent  process,  and  how  anonymity  and  confidentiality  will  be  ensured. 

6.2.6  Obtain,  set-up  and  test  equipment  and  facilities 

The  study  environment  requires  the  use  of  co  mputer  hardware/monitors  and  online  conferencing 
software  by  controllers  of  the  experiment  and  by  participants.  Brand  name  laptops  (e.g.,  Dell, 
IBM)  will  be  used  within  the  pods  and  control  room  as  described  in  section  4.1.3  Description  of 
study  environment  and  processes.  All  laptops  will  be  pre-loaded  with  the  necessary  software  and 
additional  data  collection  forms.  The  tum-aroun  d  time  to  purchase  the  co  mputer  hardware  is 
relatively  short  (i.e.,  a  month).  Research  has  already  begun  on  identify  ing  possible  online 
conferencing  software  (e.g.,  Nefsis,  Net  Meeting,  Goto  Meeting).  This  software  will  be  leased  on 
a  monthly  basis  over  the  duration  of  the  experiment  and  affords  the  benefit  of  a  n  easy  install,  no 
maintenance  and  quick  access  at  any  location  (provided  high  speed  internet  access  is  availab  le  at 
the  location).  The  testing  of  equipm  ent  can  be  done  well  in  advance  of  running  a  session.  Video 
cameras  will  also  be  purchased  for  each  pod.  The  vi  deo  cameras  will  be  a  redundant  s  ystem  to 
help  mitigate  against  any  data  loss  or  data  quality  problems  occurring  with  the  conferencing 
system.  These  cameras  can  be  m  ounted  to  the  ceiling  with  magic  arms  and  taken  down  quite 
easily  and  require  little  training  on  how  to  operate.  These  cameras  will  record  to  a  local  tape  and 
be  collected  at  the  end  of  the  session  for  analy  sis.  A  short  video  w  ill  be  made  on  how  to  use  the 
online  conferencing  software  and  viewed  by  participants  just  prior  to  the  conduct  of  each  session 
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of  the  experiment.  A  trained  technician  will  be  on  hand  at  each  location  to  answer  questions.  The 
control  room  for  all  the  sessions  as  well  as  three  additional  rooms  for  the  pods  will  be  housed  at 
the  University  of  Ottawa. 

6.2.7  Training  technical  staff,  facilitators  and  observers 

Staff  within  the  project  team  is  familiar  with  the  proposed  hardware  and  software  set-up  and 
operational  requirements.  A  technician  will  be  available  for  on-site  set-up.  Three  observers 
located  in  the  control  room,  one  for  each  pod  monitor,  will  be  trained  in  filling  out  an  observation 
log  related  to  conversations/actions  performed  by  members  of  their  assigned  pod.  The  observation 
logs  will  help  in  cueing  the  recorded  information  to  items  of  interest.  A  briefi  ng  and  debriefing 
guide  will  be  prepared  for  use  by  the  session  facilitator.  This  guide  will  include  sections  on  how 
to  introduce  the  experi  ment  to  participants,  the  sequencing  of  sim  ulation  activities/injects  and 
offer  directions  on  the  debriefing  process  and  content.  Two  facilitators  will  be  trained  in  the  use 
of  these  materials.  Initial  training  will  consist  of  several  informal  walkthroughs  of  the  experiment. 

6.2.8  Pilot  testing  of  study  environment  and  processes 

Pilot  testing  will  be  undertaken  related  to  each  independent  va  riable.  The  facilitation  of  the 
simulation  and  observation  of  participants  will  all  take  place  from  within  the  control  room. 
Students  will  be  recruited  to  act  as  participants  in  the  various  pods  for  ea  eh  pilot  test.  For 
consistency  and  time  control,  the  introduction  to  the  experiment  and  elements  of  the  debriefing 
process  will  be  presented  on  video. 

6.2.9  Recruit  subjects 

Challenges  related  to  recruitm  ent  of  study  subjects  have  been  alluded  to  in  earlier  sections  on 
descriptions  of  subjects,  pod  com  position  and  feasibility  of  experimental  plan.  Immediate  next 
steps  will  be  to  draw  up  a  master  list  o  f  potential  participants  for  the  experiment  using  existing 
networks  of  contacts.  Potential  partici  pants  will  be  identified  by  both  DRDC  and  the  University 
of  Ottawa.  The  compilation  of  t  he  initial  list  will  be  undertaken  using  the  subject  profile 
described  in  Section  4.1.2  and  session  and  pod  composition  grids  described  in  Section  4.3.23  as 
identification  and  selection  guides.  The  recruitment  pool  will  be  refreshed  on  a  continuous  basis 
over  the  conduct  of  the  experiment.  Participant  involvement  in  the  study  will  be  voluntary  and 
their  consent  will  be  obtained  as  per  ethics  requirements. 

6.2.10  Launch  experiment 

The  project  schedule  indicates  that  the  experiment  will  be  run  from  mid-September,  2010  through 
to  late  January,  2011.  Eight  sessions  wil  1  be  conducted  during  this  time  period,  with  the  specific 
dates  yet  to  be  finalized  in  order  to  accommodate  limited  participant  availability.  Several  make¬ 
up  sessions  will  also  be  planned  in  late  January  ,  2011.  These  are  not  yet  shown  on  the  schedule. 
Completion  of  the  8  sessions  b  y  late  January,  2011  provides  adequate  tim  e  for  collation  and 
analysis  of  data  and  the  preparation  of  the  report  on  findings  by  March  31,2011. 
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6.3  Project  schedule 


The  project  schedule  lists  the  start  and  end  dat  es  for  each  of  the  six  wo  rk  packages  associ  ated 
with  completing  the  entire  project.  Refer  to  Annex  F  for  details.  The  milestones  and  deliverables 
for  work  packages  1  and  2  have  been  co  mpleted.  For  the  next  work  package,  i.e.,  W3  Testing  of 
SDM  framework  in  vivo  using  sim  ulation,  specific  tasks  are  iden  tified.  These  tasks  are  gro  uped 
under  two  main  components,  qualitative  interviews  and  experiments.  The  details  about  these  tasks 
are  described  in  the  sections  above.  As  with  earlier  work  packages,  a  draft  report  of  the  findings 
will  be  submitted  for  review  one  month  prior  to  final  submission.  Work  package  3  is  scheduled  to 
start  on  June  30,  20 1 0  and  finish  on  March  3 1 ,  20 1 1 . 
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7  Application  of  findings  and  future  considerations 


This  section  outlines  potential  steps  in  applying  the  findings  from  this  study  to  the  CF  context 
and,  depending  on  the  outcome  of  the  study,  potential  avenues  of  further  investigation  that  would 
contribute  to  achieving  the  overall  TIF  objectives.  While  the  tasks  outlined  in  this  section  are  not 
within  the  current  timeframe  and  resources  allocated  to  the  present  project,  it  is  beneficial  to 
consider  throughout  the  project  (including  this  early  planning  stage)  how  the  findings  or 
knowledge  derived  from  this  study  can  be  translated,  applied,  and  transferred  to  real-world 
settings  and  situations. 


7.1  Measuring  situational  complexity 

This  study  will  result  in  refined  tools  to  assess  situation  complexity.  These  tools  will  be  grounded 
within  an  e  xtensive  theoretical  framework  and  s  ound  methodological  approach.  It  will  be 
designed  to  measure  multiple  aspects  of  any  situation  to  determine  the  extent  to  which  the  event 
at  a  particular  time  is  simple,  complicated  or  complex.  These  tools  could  be  easily  adapted  to 
other  contexts  such  a  s  training  scenario  developm  ent,  assessment  of  past  sit  uations  for  m  ore 
standardized  categorization  and  analysis,  or  planning  for  future  extreme  events.  Depending  on 
the  ease  with  which  they  can  be  imple  mented  (i.e.,  checklist  format),  they  may  even  have 
potential  to  contribute  to  situational  awareness  in  ongoing  events. 


7.2  Measuring  collaboration,  cooperation  and  coordination 

In  addition  to  tools  measuring  situation  com  plexity,  this  study  will  result  in  tools  designed  to 
assess  the  extent  to  which  collabor  ation,  cooperation  and/or  coor  dination  is  occurring  within  a 
multi-organizational  context.  Considerable  effort  will  be  expended  durin  g  the  current  study  to 
develop  tools  that  operationalize  these  concepts.  These  tools  will  be  used  by  both  external  raters 
and  participants  to  assess  the  approach  to  pr  oblem  solving  em  ployed  and  to  ensure  that 
collaboration  is  taking  place  during  the  experi  mental  sessions.  These  tools  may  have  utility 
outside  of  the  experi  ment  in  areas  such  as  evaluation  of  training  interventions,  or  asse  ssment  of 
problem  solving  post-situation  for  key  events,  and  identification  of  critical  factors. 


7.3  Further  investigate  impact  of  situation  complexity  on  problem 
solving  outcomes 

Given  the  resources  and  timeframe  for  the  current  study,  situation  complexity  has  been  restricted 
to  a  control  variable  with  the  focus  be  ing  placed  exclusively  on  com  plex  events.  The  Model 
outlined  in  Task  1  predicts  that  optimal  problem  solving  will  vary  according  to  the  complexity  of 
the  event  and  stage  in  the  response  and  event  timeline.  This  aspect  of  the  model  could  be  tested  at 
a  later  date  relatively  easily  once  the  study  infrastructure  has  been  developed  (e.g.,  scenario 
injects,  task  instructions,  pod  communication). 


DRDC  CSS  CR  201 1-31 


41 


7.4  Further  investigate  stages  of  problem  solving  and  non-linear 
problem  solving  processes 

Similar  to  situation  complexity,  the  current  study  is  restricting  the  experiment  to  focus  on  three 
problem  solving  stages  from  what  has  been  char  acterized  in  the  model  as  a  relatively  linear 
problem  solving  process  (  e.g.,  problem  definition,  solution  generation,  decision-making,  etc.). 
There  are  indications  in  the  literature  that  effective  problem-solving  is  often  not  as  linear  as 
depicted,  and  more  likely  iterative,  back-and-forth,  with  stages  skipped  al  ong  the  way  (Pa  quet, 
2009;  Edwards,  Miles  &  vonWinterfeldt,  2007).  The  non-linearity  aspect  of  problem  solving  may 
potentially  increase  as  the  process  be  comes  more  collaborative.  While  the  model  provides  an 
interpretable  approach  to  problem  solving,  the  overall  model  would  benefit  from  further  work  that 
could  characterize  the  iter  ative,  non-linear  nature  of  the  problem  solving  pr  ocess  in  co  mplex 
events.  It  is  anticipated  that  some  initial  data  may  be  collected  on  this  aspect  through  the  in-depth, 
qualitative  interviews  with  decision-makers  in  Component  One  of  the  current  study. 


7.5  Further  investigate  impact  of  various  scenarios 

To  hold  situation  complexity  as  a  control  variable  ,  the  study  will  use  one  scenario  that  has  been 
assessed  by  raters  as  most  appropriate  according  to  a  number  of  criteria  as  outlined  in  Section 
4.3. 1.2  Assessment  process  for  problem  solving  tasks.  A  similar  study  could  be  run  with  various 
scenarios  while  holding  other  variables  constant  (e.g.,  multi-organizational  context,  problem 
solving  approach).  The  addition  of  m  ultiple  scenarios  with  vary  ing  levels  of  com  plexity  (see 
point  #3  above)  could  co  ntribute  to  the  better  understanding  of  how  t  he  actual  content  of  a 
scenario  can  impact  on  the  problem  solving  outcomes.  Scenarios  could  vary  according  to  aspects 
such  as  nat  urally  occurring  disaster  s  vs.  malicious  intent,  international  vs.  do  mestic, 
infrastructures  impacted,  etc. 


7.6  Extend  the  timeline  to  include  collaboration  in  pre-event 
planning  stages 

The  scenario  for  the  current  study  will  focus  on  a  limited  portion  of  the  event  timeline  (likel  y  - 1 
to  +1).  This  is  perhaps  not  that  far  fro  m  reality  as  to  when  multiple  organizations  are  actually 
brought  together  to  address  an  extreme  event.  Ideally  the  process  of  problem  solving  would  occur 
earlier  in  the  event  timeline  during  the  pre-event  planning  stages  (-2  and  -3).  A  study  focusing  on 
the  planning,  pre-event  stages  would  provide  additional  information  a  nd  data  for  developin  g 
training  exercises  and  materials  for  how  to  effectively  collaborate,  cooperate  and  coordinate  at  the 
pre-event  stages  within  a  multi-organizational  context,  which  i  n  turn  would  likely  improve  the 
problem  solving  at  the  - 1  to  +1  stages.  This  use  of  our  design  could  obvio  usly  serve  as  well  for 
training  or  intervention  to  foster  a  collaborative  approach  to  problem-solving. 


7.7  Exploration  of  the  roles  of  leadership  vs.  stewardship  in  process 
of  problem  solving  in  the  multi-organizational  context 

While  not  within  the  scope  of  the  current  expe  riment,  a  co  mmon  question  that  persists  when 
attempting  to  have  different  organizational  structures  work  together  in  a  coordinated,  cooperative 
and  collaborative  manner  is:  who  is  in  charge ?  While  this  question  m  ay  be  a  non  sequitur  for 
those  organizations  which  rely  on  a  more  collaborative,  non-hierarchical  structure,  it  is  an 
essential  question  for  most  organizations,  particularly  those  who  operate  using  an  ICS  appr  oach 
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to  addressing  situations.  Given  the  challenges  in  addressing  this  question  in  a  collaborative 
endeavour,  it  can  create  frustration  and  unease  among  and  within  organizations  who  are  asked  to 
think  about  leadership  an  d  “who  is  in  charge”  in  a  different  manner  when  participating  in  a 
collective  effort.  Recent  work  on  collabora  tion,  governance,  and  decision-m  aking  where 
traditional  notions  of  “leadership”  are  reconceptualised  as  “stewardship”  could  be  useful  in 
developing  approaches  that  are  more  congruent  with  collaboration,  and  yet,  acceptable  to  those 
coming  from  more  traditional  leadership  models  (Paquet,  2009). 


7.8  Translation  of  tasks  into  training  materials  for  learning  and 
practicing  collaborative  techniques  in  multi-organizational 
settings  involving  complex  situations 

One  key  independent  var  iable  is  appr  oach  to  problem  solving.  Considerabl  e  effort  will  be 
expended  in  developing  specific  tasks  that  will  result  in  either  coordination  or  collaboration 
within  and  between  vario  us  pod  compositions.  These  tasks  could  be  translated  into  training 
materials  for  learning  and  practicing  techniques  in  coordination,  cooperation  and  collaboration. 


7.9  Transfer  experimental  setting  into  prototype  training  setting 

Given  the  in  vivo  nature  of  the  experimental  setting,  it  could  be  transformed  from  an  experimental 
setting  into  a  protot  ype  training  setting  which  w  ould  apply  the  various  findings  from  the  study 
into  a  training  context  using  many  of  the  sa  me  materials,  along  with  pot  entially  additional 
considerations  such  as  additional  scenarios,  extended  timelines  and  revised  tasks  (s  ee  sections 
7.5,  7.6 ,  and  7.8  above). 

7.10  Test  prototype  training  with  recruits/trainees  and  develop 
materials  and  recommendations  for  training  of  members  of 
operational  community 

Once  prototype  training  modules  had  been  developed,  they  could  go  through  a  rigorous  training 
evaluation  procedure  using  various  groups  ofCF  recruits/trainees,  along  with  participants  from 
various  federal,  provincial  and  community  agencies.  Prototype  training  materials  and  tools  would 
be  revised  accordingly  ,  and  recommendations  developed  for  im  plementation  within  the 
operational  community. 
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Annex  A  Task  1  Report  Figures 
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Figure  7:  Modifying  variables  of  power,  resources  and  information  (adapted  from  Crosby  & 

Bronson,  2005) 
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Figure  8:  Model  of  inter-organizational  problem  solving 
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Figure  9:  Management  approaches  by  time  phase 


Figure  10:  Representation  of  interdependence  of  coordination,  cooperation,  and  collaboration 
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Figure  11:  Three  factors  contributing  to  situation  complexity 


Figure  12:  Representation  of  coordinated  organizations 


Figure  13:  Representation  of  cooperating  organizations 
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Figure  14:  Representation  of  collaborating  organizations 


DRDCCSSCR  2011-31 


49 


Annex  B  Preliminary  Guidelines  for  Interviews 


Background 

This  part  of  the  research  project  will  explore  the  narrative  of  key  decision  makers  of  major  events. 
Specifically  we  wish  to  explore  how  they  managed  complex  crisis  that  involved  the  participation 
of  multiple  organizations.  The  research  project,  led  by  Dr.  Louise  Lemyre  from  the  Institute  for 
Population  Health  at  the  University  of  Otta  wa,  is  funded  by  the  Defence  Research  and 
Development  Canada  (DRDC).  This  phase  of  th  e  project  will  invol  ve  20  to  30  i  ndividual 
interviews  across  the  region  of  Ottawa,  Montreal  and  Toronto,  the  location  of  the  interviews  will 
be  agreed  according  upon  the  intervie  wee' s  convenience  to  assure  a  confide  ntial  environment 
since  the  nature  of  the  responses  m  ight  imply  delicate  organizational  topics.  The  interviews  will 
be  conducted  by  a  senior  interviewer  and  a  note  take  r,  and  be  tap  e  recorded  so  that  the  y  can  be 
transcribed.  The  responses  will  be  f  urther  analyzed  by  the  research  team  in  keeping  the 
confidentiality  agreement. 

The  objectives  of  the  interviews  are  to  explor  e  the  key  stakeholders’  perception  of  multi 
organization  problem  solving  during  complex  crisis.  The  analy  sis  of  the  resu  Its  will  help  to 
identify  the  multiple  dimensions  intertwined  in  th  e  response  of  such  events,  such  as  complexity, 
managerial  processes,  outcomes  and  multi  organizational  structures  and  environment.  Given  the 
phenomenological  approach  of  the  inte  rviews,  the  objective  is  to  understand  t  he  crisis  events 
from  the  perspective  of  the  interv  iewee,  therefore  no  right  or  wrong  answer  s  are  to  be  expected. 
The  interview  will  be  guided  following  an  adaptation  from  the  “Critical  Decision  Method” 

The  treatment  of  the  data  gathered  from  the  interview  will  be  as  follows: 

•  All  the  audio  recordings  will  be  safely  stored  in  the  facilities  of  the  GAP-Sante  Laboratory 

•  The  information  from  the  interviews  will  be  transcribed  verbatim 

•  The  transcribed  information  will  be  analy  zed  using  the  NVi  vo  software  that  process 
qualitative  information. 

The  coding  of  data  will  be  performed  in  a  cascading  manner  by  two  different  researchers,  then  the 
coding  results  will  be  compared.  If  a  consensus  is  not  reached,  the  coding  scheme  will  be  refined 
or  tuned  up.  Inter-rater  reliability  will  be  assessed. 
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Interview  Questions 

Probes  will  not  necessarily  be  asked  in  the  following  order  and  if  the  question  has  already 
been  answered  the  probe  will  not  be  used.  Questions  will  be  context  specific  on  interviewees’ 
recollection  of  the  event. 

Introductory  Questions 

•  What  organization  do  you  work  for  now  and  what  organization  did  you  work  for  during  the 
situation? 

•  What  is  your  role  currently? 

•  What  was  your  role  during  the  situation? 

•  How  long  had  you  been  in  this  role  prior  to  the  crisis? 

1.  Will  you  give  me  a  quick  run  through  the  main  problems  during  the  incident,  where  multiple 
organizations  were  involved? 

a.  Could  you  describe  each  of  these  situations? 

b.  What  circumstances  lead  up  to  each  of  these  situations? 

c.  What  information  did  you  need  in  each  of  these  situations? 

d.  Who  was  involved  in  solving  each  of  these  situations? 

e.  Which  problems  were  solved? 

f.  Were  there  any  special  facilitators  involved? 

i.  How  did  these  special  facilitators  come  about? 

g.  Did  you  feel  that  anyone  was  missing  from  the  problem  solving  process? 

Organizational  Problem  Solving 

2.  How  were  the  most  important  issues  solved  during  the  event? 

a.  Why  were  the  issues  solved  this  way? 

b.  What  type  of  power  dynamics  existed  when  trying  to  find  a  solution  to  the  incident? 

i.  before  the  event 

ii.  during  the  event 

iii.  after  the  event 
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c.  Did  someone  or  some  organization  take  control  when  making  decisions  or  was  there 
more  equality  in  the  decision  making  process? 

i.  What  observations  lead  you  to  this  conclusion? 

d.  What  was  the  final  decision? 

Team  and  task 

3.  Did  you  originally  expect  the  problem  solving  process  to  occur  as  it  did?  Why  or  why  not? 

a.  Did  any  part  of  the  problem  solving  process  surprise  you? 

b.  Do  you  usually  prefer  a  different  problem  solving  method  than  the  one  you  engaged 
in? 

c.  Did  you  understand  the  situations  in  a  similar  way  to  those  you  were  working  with? 

d.  Did  you  expect  other  people  to  participate  at  the  pace  that  they  did? 

e.  Did  you  feel  that  the  individuals  you  were  working  with  thought  like  you? 

f.  Did  you  feel  that  the  individuals  you  were  working  with  understood  you? 

Problem  Solving  Process 

4.  Can  you  describe  the  specific  problem  solving  steps  that  was  taken  in  each  situation? 

a.  How  was  the  problem  identified? 

b.  How  was  the  problem  defined? 

c.  How  was  the  solution  generated? 

d.  How  was  a  decision  reached? 

e.  How  was  the  solution  implemented? 

f.  Did  you  receive  feedback  on  the  success  or  failure  of  your  implemented  decisions? 

g.  Were  you  able  to  use  this  feedback  in  future  problem  solving? 

Sharing 

5.  Did  you  perceive  sharing  between  those  involved  in  the  incident?  And  If  so,  why  was  there 
sharing  and  when  did  it  occur? 

a.  Did  you  perceive  information  sharing  between  those  involved  in  the  incident? 

i.  Did  you  perceive  individuals  volunteering  information  willingly  or  needing  to  be 
asked  to  do  so? 
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b.  Did  you  perceive  activity  sharing  between  those  involved  in  the  incident? 

i.  Did  you  perceive  individuals  volunteering  to  share  activities  or  did  they  need  to 
be  asked  to  do  so? 

c.  Did  you  perceive  resources  (human  resources,  economic,  or  infrastructure)  sharing 
between  those  involved  in  the  incident? 

i.  Did  you  perceive  individuals  volunteering  resources  willingly  or  did  they  need  to 
be  asked  to  do  so? 

d.  Did  you  perceive  power  sharing  between  those  involved  in  the  incident? 

i.  Did  you  perceive  individuals  volunteering  power  willingly  or  did  they  need  to  be 
forced  to  do  so? 

ii.  Did  you  perceive  any  power  struggles?  Please  describe  it  to  me. 

Goals 

6.  How  were  goals  discussed,  created,  and  prioritized  concerning  the  situation? 

a.  Did  you  or  did  anyone  you  were  working  with  follow  personal  goals? 

b.  Did  you  or  did  anyone  you  were  working  with  follow  their  organization’s  goals? 

c.  Did  you  and  those  you  were  working  with  combine  your  individual  organization’s 
goals  into  a  larger  goal? 

d.  Did  you  and  those  you  were  working  with  create  a  new  common  goal? 

e.  Did  the  goals  change  as  the  situation  evolved? 

Multi-organizational  environment 

7.  From  your  perspective,  what  was  the  effect  of  multiple  organizations  participating  in  joi  nt 
problem  solving? 

a.  How  did  you  perceive  the  level  of  group  cohesion? 

i.  How  did  you  perceive  the  level  of  group  cohesion  between  individuals  from  the 
same  organization?  And  between  individuals  from  different  organizations? 

b.  What  was  your  perception  of  the  level  of  conflict  between  individuals  from  the  same 
organization?  And  between  individuals  from  different  organizations? 

c.  How  was  conflict  managed  between  individuals  from  the  same  organizations?  And 
between  individuals  from  different  organizations 

d.  What  was  your  perception  of  the  level  of  agreement  between  individuals  from  the 
same  organizations?  And  between  individuals  from  different  organizations? 
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Complexity 

Uncertainty 

8.  How  did  you  experience  uncertainty  surrounding  these  situations? 

a.  Did  you  ever  deal  with  similar  situations  prior  to  this  event? 

b.  Did  you  anticipate  or  plan  for  this  problem  beforehand? 

c.  Did  you  feel  that  the  information  available  was  sufficient  to  make  a  decision? 

d.  Were  there  new  organizations  or  partners  involved  that  you  had  never  dealt  with 
before?  And  if  so,  which  organizations  or  partners? 

e.  From  your  perspective  was  the  problem  easy  to  understand? 

f.  In  your  perspective,  did  the  context  surrounding  the  situations  constantly  change? 
ii.  What  changed,  when  did  it  change,  and  how  did  this  make  an  impact? 

Workload 

9.  Could  you  discuss  how  the  workload  changed  across  situations  during  the  incident?  And  why 
did  these  changes  occur. 

a.  What  needed  to  be  done  to  solve  the  main  situations? 

b.  At  some  point,  did  you  feel  overwhelmed  with  all  the  things  you  needed  to  do? 

Personal  Perception 

10.  What  was  your  level  of  participation  in  each  situation? 

a.  Were  you  satisfied  with  the  problem  solving  process? 

b.  Do  you  think  the  timing  of  the  problem  solving  process  was  appropriate? 

c.  Were  you  satisfied  with  the  level  of  information,  activity,  resource  and  power  sharing 
during  the  problem  solving  process? 

d.  How  do  you  perceive  the  quality  of  the  decision? 

e.  What  (other)  difficulties  did  you  have? 

f.  Is  there  anything  else  you  would  like  to  discuss? 
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Annex  C  Situation  complexity  rating  scales 


SCALE 

Less  complex  More  complex 

1.1 

Factor  A: 
Event 
impact 

0 

1 

2 

3 

4 

1.1.1 

Scope  of 
impacts 

1.1. 1.1 

Tier  1 :  Direct  effects 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1. 1.2 

Tier  2:  Effects  on 
essential  services 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1. 1.3 

Tier  3:  Effects  on 
societal  functions 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1.2 

Severity  of 
impacts 

1. 1.2.1 

Number  of  fatalities/ 
casualties 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1. 2.2 

Amount  of  physical 
damage 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1. 2.3 

Amount  of 
psychological 
trauma 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1. 2.4 

Degree  of  dispersion 
-  people  /  place 

None 

Community 

Regional 

National 

International 

1.1. 2.5 

Degree  of  impact  on 
response 
organizations 

None 

Minimal 

Moderate 

Severe 

Catastrophic 

1.1.3 

Timing  of 
impacts 

1. 1.3.1 

Threat  /  warning 
only 

Yes 

No 

1. 1.3.2 

Sudden  /  short 

No 

Yes 

1. 1.3.3 

Sustained  /  lengthy 

No 

Yes 

1.1. 3.4 

Recurring  / 
intermittent 

No 

Yes 

1.1.4 

Involvement 
of  media 

1. 1.4.1 

Degree  /  length  of 
coverage 

None 

Minimal 

Moderate 

Extensive 

Excessive 

1.1. 4.2 

Impact  on 
perceptions 

None 

Minimal 

Moderate 

High 

Very  high 

1.1. 4.3 

Access  to  official 
sources  of 
information 

Very  large 
number  of 
sources  /  easy 
to  access 

Many  sources 
/  Somewhat 
easy  to  access 

Moderate 
number  of 
sources  /  access 
neither  difficult 
nor  easy 

Few  sources 
/  Somewhat 
difficult  to 

access 

No  sources  / 

no  access 
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SCALE 

Less  complex  More  complex 

0 

1 

2 

3 

4 

1.1. 4.4 

Utilization  of 
social  media 

Very 

effective 

utilization 

Effective 

utilization 

Somewhat 

effective 

utilization 

Ineffective 

utilization 

No  utilization 

1. 1.4.5 

Degree  of  public 
outrage  /  fear 

None 

Minimal 

Moderate 

High 

Very  high 

1.1.5 

Political 

processes 

1. 1.5.1 

Number  of 
jurisdictions 

None 

Low 

Moderate 

High 

Very  High 

1. 1.5.2 

Type  of 
jurisdictions 

Local  only 

Local  and 
regional 

Local,  regional 
and  national 

Local, 
regional, 
national  and 
military 

Local, 

regional, 

national, 

military. 

international 

1. 1.5.3 

Level  of 
jurisdictions 

Commun¬ 

ity 

Regional 

National 

Military 

International 

1.2 

Factor  B: 
Uncertainty 

1.2.1 

Novelty  of 
situation 

1.2.1. 1 

Hazard  type  - 
CBRNE,  other 

Not 

CBRNe 

CBRNe 

1.2. 1.2 

Cause:  unknown, 
malicious, 
accidental 

Known 

cause, 

accidental 

Known 

cause, 

malicious 

Unknown  cause, 
accidental 

Unknown 

cause, 

unknown  if 
accidental  or 
malicious 

Unknown 

cause, 

malicious 

1.2. 1.3 

Degree  of  prior 
experience 

Very  high 

High 

Moderate 

Low 

None 

1.2. 1.4 

Controllability 

Very  high 

High 

Moderate 

Low 

None 

1.2. 1.5 

Latency 

No 

Yes 

1.2.2 

Anticipation 
&  planning 

1. 2.2.1 

Developed  plans 
-  known/ 
unknown  hazards 

No 

Yes 

1.2.2.2 

Rehearsed  plans 

Plans  very 
well 

rehearsed 

Plans  well 
rehearsed 

Plans  somewhat 
rehearsed 

Very  little 
rehearsal  of 
plans 

No  rehearsal  of 
plans 

1.2.2. 3 

Shared  plans 

Extensive 
sharing  of 
plans 

Plans 

shared 

Some  sharing  of 
plans 

Very  little 
sharing  of 
plans 

No  sharing  of 
plans 

1.2.3 

Lack  of  data 
&  information 

1.2.3. 1 

Access  to 
information 

Easy  to 
access 

Somewhat 
easy  to 
access 

Access  neither 
difficult  nor  easy 

Somewhat 
difficult  to 

access 

No  access 

1. 2.3.2 

Accuracy/ 
completeness  of 
information 

Very 

accurate 

and 

complete 

Accurate 

and 

complete 

Partially  accurate 
and  partially 
complete 

Inaccurate  and 
incomplete 

Very 

inaccurate  and 
incomplete 
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SCALE 

Less  complex  More  complex 

0 

1 

2 

3 

4 

1. 2.3.3 

Comprehensibility  of 
information 

Very  easy 
to 

understand 

Easy  to 
understand 

Somewhat 
easy  to 
understand 

Difficult  to 
understand 

Very  difficult 
to  understand 

1. 2.3.4 

Credibility  of 
information  sources 

Highly 

credible 

Somewhat 

credible 

Credibility  not 
known 

Credibility 

suspect 

No  credibility 

1. 2.3.5 

Volume  of  relevant 
/extraneous  data 

None 

Low 

Moderate 

High 

Very  high 

1. 2.3.6 

Speed  of  information 
flow 

Very  fast 

Fast 

Moderate 

Slow 

Very  slow 

1.2.4 

New  organizations 
&  partners 

1. 2.4.1 

Traditional 
organizations  & 
partners 

Yes 

No 

1. 2.4.2 

Non-traditional 
organizations  & 
partners 

Yes 

No 

1. 2.4.3 

Emergent 

organizations 

No 

Yes 

1.2.5 

Rapidly  changing 
context 

1. 2.5.1 

Nature  /  degree  of 
change 

Very 

positive 

Positive 

Neutral 

Negative 

Very  negative 

1. 2.5.2 

Speed  of  change 

Very  slow 

Slow 

Moderate 

Fast 

Very  fast 

1. 2.5.3 

Consequences  of 
change  - 
actual/potential 

Very 

positive 

Positive 

Neutral 

Negative 

Very  negative 

1. 2.5.4 

Impacts  of 
interventions 

None 

Low 

Moderate 

High 

Very  high 

1.2.6 

Flexibility  of 
interpretive 
frameworks 

1. 2.6.1 

Imposed  /  unilateral 
process 

No 

Yes 

1. 2.6.2 

Common  /  shared 
process 

Yes 

No 

1. 2.6.3 

Degree  of  creativity  / 
improvisation 

None 

Low 

Moderate 

High 

Very  high 

1.3 

Factor  C: 
Vulnerability 
(Resiliency) 

1.3.1 

Economic 

development 

1.3.1. 1 

Amount  of  resources 

Very  high 

High 

Moderate 

Low 

None 

1.3. 1.2 

Diversity  of 
economic  activity 

Very  high 

High 

Moderate 

Low 

None 

1 .3.1.3 

Distribution  of 
wealth 

Very  low 
level  of 
inequity 

Low  level  of 
inequity 

Moderate  level 
of  inequity 

High  level  of 
inequity 

Very  high 
level  of 
inequity 
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SCALE 

Less  complex  More  complex 

0 

1 

2 

3 

4 

1.3. 1.4 

Economic  stability 

Very  strong 

Strong 

Moderate 

Weak 

None 

1.3.2 

Social  capital 

1. 3.2.1 

Socio-demographics 
/  at-risk  populations 

None  at-risk 

Few  at-risk 

Moderate 
number  at-risk 

Many  at-risk 

Very  many  at- 
risk 

1. 3.2.2 

Social  support 

Very  strong 

Strong 

Moderate 

Weak 

None 

1.3.2. 3 

Social  identity 

Very  strong 

Strong 

Moderate 

Weak 

None 

1. 3.2.4 

Citizen  participation 

Very  high 

Strong 

Moderate 

Low 

None 

1. 3.2.5 

Sense  of  community 

Very  high 

Strong 

Moderate 

Low 

None 

1. 3.2.6 

Attachment  of  place 

Very  high 

Strong 

Moderate 

Low 

None 

1.3.3 

Community 

competence 

1.3.3. 1 

Political  stability 

Very  strong 

Strong 

Moderate 

Weak 

None 

1.3.3.2 

Leadership  skills 

Very  strong 

Strong 

Moderate 

Weak 

None 

1.3.3. 3 

Level  of  education 

Very  high 

High 

Moderate 

Low 

None 

1.3.3.4 

Prior  experience 

Very  high 

High 

Moderate 

Low 

None 

1.3. 3. 5 

Community 
partnerships  - 
extemal/intemal 

Very  strong 

Strong 

Moderate 

Weak 

None 

1.3.4 

Information 

& 

communicatio 

n 

1.3.4. 1 

Number  of 
information  sources 

Very  large 
number  of 

sources 

Many  sources 

Moderate 
number  of 

sources 

Few  number  of 

sources 

None 

1. 3.4.2 

Public  vs.  expert 
perceptions 

Public 
perceptions 
aligned  with 
expert 
perceptions 

Public 
perceptions 
somewhat 
aligned  with 
expert 
perceptions 

Public 

indifference  to 
expert 
perceptions 

Public 
perceptions 
somewhat 
aligned  with 
expert 
perceptions 

Public 

perceptions  not 
at  all  aligned 
with  expert 
perceptions 

1.3.4.3 

Trusted  sources  of 
information 

Yes 

No 

1. 3.4.4 

Efficacy  of  risk 
messaging 

Very  effective 

Effective 

Neutral 

Ineffective 

Damaging 

1. 4.4.5 

Communication 

infrastructure 

Very  strong 

Strong 

Moderate 

Weak 

None 
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SCALE 

Less  complex  More  complex 

0 

1 

2 

3 

4 

1.3.5 

Other 

infrastructure 

1.3.5. 1 

Degree  of 
interdependence  re: 
cascading  failures 

Very  low 

Low 

Moderate 

High 

Very  high 

1.3. 5.2 

Presence  of 
redundant  systems 

Very  high 

High 

Moderate 

Low 

None 

1.3. 5.3 

Distribution 
networks  /  supply 
chains 

Very  high 

High 

Moderate 

Low 

None 
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Annex  D  Three-way  Analytic  Grid 


The  draft  actions  described  below  are  related  to  a  potential  cyber  attack  scenario. 


PROBLEM  SOLVING  STAGES 

Problem 

Identification 

Awareness  of 
issue 

Making 

assumptions 

Data  gathering 

Agreement  that 
issue  exists 
Identification  of 
potential  impacts 

Problem 

Definition 

Decomposing  the 
issue 
Verifying 
assumptions 
Setting 
boundaries 

Root  cause 
analysis 

Data 

interpretation 
Framing  the 
problem 

Solution 

Generation 

Developing 

options 

Evaluating 

options 

Prioritizing 

options 

Decision  Making 

Choosing  an 
option 
Qualifying 
decision 
Communicating 
decision 

Solution 

Implementation 

Committing  to 
decision 

Assigning  roles 

Assigning 

accountability 

Assigning 

resources 

Feedback: 

Success/Failure 

Gathering  data 

Monitoring 

consequences 

Changing  plans 

Communicating 

changes 

Re-allocating 

resources 

Approach 

Time  Assets 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Coordination 

Pre 

Information 

Share  standard 

EM  plans 

Share  case  studies 
of  cyber-attacks 

Consult  other 
organizations 
when  weighing 
options 

Inform  other 
organizations  of 
decision 

Practice  standard 
EM  plans 

No 

"A  process  of 
communication, 
planning  and  sharing 
of  resources,  risk  and 
rewards  for  purposes 
of  efficiency  and 
effectiveness  in 
achieving  the 
complementary  goals 
of  the  parties  involved" 
(Taylor-Powell  et  al., 
1998)  Emphasis  on: 
avoidance  of 
duplication;  efficiency 
of  service,  sharing 
primarily  information, 
parallel  activities, 
independent  decision 
making. 

(-3  to  -2) 

Activities/ 

Resources 

No 

No 

No 

No 

No 

No 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 

During 

Information 

Discuss  news 
reports  and  other 
official  info 

Share  possible 
anti-virus 
solutions 

Organize  own 
recall  of  systems, 
weigh  options 

Share  possible 
viral  solutions  and 
recall  plans  with 
other  orgs. 

Begin  recall,  and 
offer  their  plan  to 
other  orgs. 

Monitor  amount  of 
returned  systems, 
and  impact  on  virus 

(-1  to 

+i) 

Activities/ 

Resources 

No 

No 

No 

No 

No 

No 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 

Post 

Information 

Discuss  how  to 
prevent 

recurrence 

Discuss  lingering 
/  latent  issues 

Create  resiliency 
plan  for  future 
events 

Effect  a  policy 
change  for  safer 
importing  of 
computer  parts 

Install  better  anti¬ 
virus 

Discuss  lessons 
learned 

(+2  to 
+3) 

Activities/ 

Resources 

No 

No 

No 

No 

No 

No 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 
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PROBLEM  SOLVING  STAGES 

Problem 

Identification 

Awareness  of 
issue 

Making 

assumptions 

Data  gathering 

Agreement  that 
issue  exists 
Identification  of 
potential  impacts 

Problem 

Definition 

Decomposing  the 
issue 
Verifying 
assumptions 
Setting 
boundaries 

Root  cause 
analysis 

Data 

interpretation 
Framing  the 
problem 

Solution 

Generation 

Developing 

options 

Evaluating 

options 

Prioritizing 

options 

Decision  Making 

Choosing  an 
option 
Qualifying 
decision 
Communicating 
decision 

Solution 

Implementation 

Committing  to 
decision 

Assigning  roles 

Assigning 

accountability 

Assigning 

resources 

Feedback: 

Success/Failure 

Gathering  data 

Monitoring 

consequences 

Changing  plans 

Communicating 

changes 

Re-allocating 

resources 

Approach 

Time  Assets 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Cooperation 

Pre 

Information 

Share  standard 

EM  plans 

Share  case  studies 
of  cyber-attacks 

No 

No 

Practice  standard 
EM  plans 

No 

"A  process  where 
parties  with  similar 
interests  plan  together, 
negotiate  mutual  roles, 
and  share  resources  to 
achieve  joint  goals,  but 
maintain  separate 
identities  (Taylor- 
Powelletal.,  1998)." 
Emphasis  on: 
coordinating  existing 
resources,  filling 
resource  gaps,  creating 
new  organizations  to 
fill  gaps,  greater 
sharing  of  information 
plus 

activities/resources, 
more  interdependence 
(problem  definition, 
solution 

implementation)  more 
joint  deciions  making; 
able  to  achieve  more 
than  usual  tasks  -  have 
greater  impact 

(-3  to  -2) 

Activities/ 

Resources 

P-Ram  Communi¬ 
cations  Develops 
anti-virus 
software 

Memorandum  of 
Understanding  / 
Creating 
Partnerships 
between  orgs. 

P-Ram 

Communications 
distributes  anti¬ 
virus  software 

Asset  map  to 
assess  possible 
gaps  in  essential 
services 

Form  Roundtable 
to  share  info,  and 
practice  plans, 
and  to  distribute 
anti-virus 

No 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 

During 

Information 

Shared  expert 
opinions  and  data 

Determine  extent 
of  effect 

Create  appropriate 
risk  messaging 

Choose  trusted 
sources  to  deliver 
message 

Hold  a  press 
conference 

Discuss  if  medium 
was  appropriate, 
reached  entire  target 
audience 

(-1  to 
+1) 

Activities/ 

Resources 

Contact  other  orgs 
via  phone,  email, 
fax,  to  collect 
their  current  data 

Assess  resource 
gaps 

Discuss  how 

resources  are  to 
be  deployed  / 
shared 

P-RAM  IT 
technicians  to  go 
onsite,  to  collect 
systems,  effect 
anti-virus  repairs 

Coordinate  efforts 
between  orgs.  for 
solution  delivery 

Did  technicians 
reach  target  groups? 
Discuss  possible 
shortcomings 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 

Post 

Information 

Discuss  possible 
recurrence 

Discuss  lingering 
/  latent  issues 

Create  resiliency 
plan  for  future 
events 

Effect  a  policy 
change  for  safer 
importing  of 
computer  parts 

Install  better  anti¬ 
virus 

Town  hall  meeting 
to  discuss  results 
and  seek  input  for 
future  policy 
changes 

(+2  to 
+3) 

Activities/ 

Resources 

How  many 
systems  need  to 
be  rebuilt? 

Possible  short 
term  solutions? 

Reconstruction  of 
corporate  systems 

Who  will  pay  for 
the  replacement 
systems?  Who 
will  do  the  work? 

Coordination  of 
comprehensive 
recovery  and 
reconstruction 
plan 

Conduct  or  fund 
studies  to  monitor 
interventions  at  the 
recovery  / 
reconstruction 
stages 

Power/ 

Authority 

No 

No 

No 

No 

No 

No 
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PROBLEM  SOLVING  STAGES 

Problem 

Identification 

Awareness  of 
issue 

Making 

assumptions 

Data  gathering 

Agreement  that 
issue  exists 
Identification  of 
potential  impacts 

Problem 

Definition 

Decomposing  the 
issue 
Verifying 
assumptions 
Setting 
boundaries 

Root  cause 
analysis 

Data 

interpretation 
Framing  the 
problem 

Solution 

Generation 

Developing 

options 

Evaluating 

options 

Prioritizing 

options 

Decision  Making 

Choosing  an 
option 
Qualifying 
decision 
Communicating 
decision 

Solution 

Implementation 

Committing  to 
decision 

Assigning  roles 

Assigning 

accountability 

Assigning 

resources 

Feedback: 

Success/Failure 

Gathering  data 

Monitoring 

consequences 

Changing  plans 

Communicating 

changes 

Re-allocating 

resources 

Approach 

Time  Assets 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Collaboration 

Pre 

Information 

Social  Media  / 
Internet 

Surveillance  early 
warning  systems 

Flexible 
interpretive 
framework  used  / 
problems  not 
defined  by  assets 
available 

Wide  variety  of 
stakeholders 
included  in 
solution 
generation 

Decision  making 
conducted  in  a 
transparent 
manner 

Plans  shared  and 
roles  clarified 
with  all  interested 
parties 

No 

"A  process  through 
which  parties  who  see 
different  aspects  of  the 
problem  can  explore 
constructively  their 
differences  and  search 
for  (and  implement) 
solutions  that  go 
beyond  their  own 
limited  vision  of  what 
is  possible"  (Taylor- 
Powelletal.,  1998). 
Emphasis  omshared 
responsibility  and  a 
greater  inter 
dependency;  shared 
vision  with  partitioned 
mandates;  voluntary 
participation  and 
consent;  multiple 
perspectives 
(inclusive);  shared 
products  and 
credit. (Taylor-Powell 
&  Rossing,  2009) 

(-3  to  -2) 

Activities/ 

Resources 

Skills  inventory 
created  to 
determine  assets 
ahead  of  time 

Structural 
weakness 
identified  by 
looking  at  the 
issues  /  gaps 
holistically 

At-risk 
populations 
considered  when 
prioritizing 
options 

Option  chosen 
based  on 
projected  needs 
rather  than 
immediately 
available 

resources 

Memorandums  of 
understanding 
used  to  outline 
roles 

No 

Power/ 

Authority 

Issues  determined 
through  verifiable 
evidence 

Creation  of  shared 
vision  among 
multi-stakeholders 

Priorities  based  on 
local  knowledge  / 
evidence  /  deferral 
to  experts 

Public 

consultation  in 
decision  making 
process 

Final  decisions 
elicit  public  trust 

No 

During 

Information 

Information 
shared  with  other 
concerned 
organizations  in  a 
transparent 
manner  (within 
security 
constraints) 

Flexible 
interpretive 
framework  used  / 
looking  at  the 
problem  from 
different 
perspectives 

Utilize  crowd 
sourcing 
technology  to 
make  sense  out  of 
data  quickly 

Effective  risk 
messaging  (early 
and  often) 

Combined 
expertise  from 
public  and  private 
sector  to  deliver 
message 

No  fault  reporting 
used  to  gain 
knowledge  from  the 
ground  up  on 
intervention  efficacy 

(-1  to 
+1) 

Activities/ 

Resources 

Examine  data 
with  experts  from 
several  different 
fields  of  IT  and 
EM 

Determine  the 
flow  and  amount 
of  data  to  be 
process  in  relation 
to  available  man¬ 
power 

Consider  the 
possible  networks 
in  the  community 
(i.e./  phone  trees, 
university 
computer  labs) 

Contact  the 
identified 
organizations,  and 
have  them  begin 
processing  data 

Utilizing 
emergent 
organizations  and 
crowd  sourced 
possibilities  for 
virus  control 

Use  analytics  to 
determine  rate  of 
public  involvement 

Power/ 

Authority 

Convene  heads  of 
private  IT  sector 
and  government 
officials 

Discuss  the  range 
of  damage  / 
municipal  / 
provincial  / 
federal 

Decide  which 
systems  need  to 
be  dealt  with  first 
(ie.  hospital  a 
priority) 

Decisions 
determined  by 
local  needs 

Improvisation 
used  in 
establishing  a 
governing 
framework  for  the 
mass  recall 

Speed  and  ease  of 
decision  making 
cycle  determines 
success  of  shared 
authority  structure 

Post 

Information 

Data  gathered 
from  affected 
populations 

Flexible 
interpretive 
framework  used  / 
looking  at  the 
problem  from 
different 
perspectives 

Establish  a  set 
meeting  time  for 
possible  exercises 
and  drills  amongst 
all  the  identified 
orgs. 

Contact  the 
identified 
organizations  with 
monthly  updates 

Hold  drills, 
meetings, 
roundtables  and 
exercises 

Evaluate  the 
possible  success  or 
failure  of  the  shared 
information  amongst 
many  levels  of 
government  and 
private  sector 
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PROBLEM  SOLVING  STAGES 

Problem 

Identification 

Awareness  of 
issue 

Making 

assumptions 

Data  gathering 

Agreement  that 
issue  exists 
Identification  of 
potential  impacts 

Problem 

Definition 

Decomposing  the 
issue 
Verifying 
assumptions 
Setting 
boundaries 

Root  cause 
analysis 

Data 

interpretation 
Framing  the 
problem 

Solution 

Generation 

Developing 

options 

Evaluating 

options 

Prioritizing 

options 

Decision  Making 

Choosing  an 
option 
Qualifying 
decision 
Communicating 
decision 

Solution 

Implementation 

Committing  to 
decision 

Assigning  roles 

Assigning 

accountability 

Assigning 

resources 

Feedback: 

Success/Failure 

Gathering  data 

Monitoring 

consequences 

Changing  plans 

Communicating 

changes 

Re-allocating 

resources 

Approach 

Time 

Assets 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

Actions 
Demonstrating 
Asset  Sharing 

(+2  to 
+3) 

Activities/ 

Resources 

Examine  data 
regarding 
networked 
systems  and  their 
speed  and  ease  of 
use  during  the 
event 

Define  how  these 
networks, 
antiviruses  and 
firewalls  can  be 
expanded,  better 
utilized 

Use  the  emergent 
groups  presented 
during  the  impact 
phase  to  help 
strengthen  the 
network 

Contact  the 
organizations,  and 
the  public.  Invite 
them  to  join  the 
effort  of  increased 
security 

Create  a  joint  task 
force  for  future 

events 

Appoint  board  of 
governors  to  assess 
efficacy. 

Power/ 

Authority 

Examine 
information 
regarding  the 
shared  scope  of 
effects  and 
responsibilities 

Consider,  can  the 
public  be  both 
victim  and  rescuer 
in  a  cyber-attack? 

Public  engaged  in 
the  solution 
generating 
process 
(community 
leaders  brought  to 
the  table) 

Public 

consultation  in 
decision  making 
process 

Resources 
allocated 
according  to  local 
needs 

Share  blame  / 

success 
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Annex  E  Considerations  for  Dependent  Variables  -  Problem 
Solving  Processes  and  Outcomes 


Problem  Solving  Processes 


Variable 

Name 

Considerations 

Type  of  Measure  & 
Frequency 

Required  Actions 

Satisfaction 

with 

problem 

solving 

process 

•  Need  to  ensure  that  this  is 
separate  from  problem  solving 
outcome  (e.g.,  decision)  - 
emphasize  the  process 

•  Bi-directional  Likert-type  scale 
with  well  anchored  ends 

•  Very  Dissatisfied,  Dissatisfied, 
Neutral,  Satisfied,  Very  Satisfied 

•  Key  components  would  include 
satisfaction  with: 

•  overall  process 

•  time  required 

•  participation  from  team 

•  opportunities  for  you  to 
provide  input 

•  difficulty  of  task 

•  information/instructions 
received 

•  Self  report  (pen  and 
paper) 

•  Quick  measure  of  key 
components  after  each 
task  if  possible  (2X) 
(in-session) 

•  One  overall  for  session 
(debrief) 

Will  need  to  develop 
in-house: 

•  Self  report  measure 

Level  of 
participation 
(intra-pod) 

Would  be  rated  on  a  number  of 

components. 

•  Pattern,  total  &  proportion  of 
minutes  talking  by  individual 

•  Pattern,  total  &  proportion  of 
minutes  individual  is  engaged 
with  group  (talking  or  active 
listening) 

•  Pattern,  total  &  proportion  of 
minutes  of  individual  is 
disengaged  from  (independent 
work,  reading) 

•  Pattern,  total  &  proportion  of 
solicited 

suggestions/ideas/alternatives 
provided  by  individual 

•  Pattern,  total  &  proportion  of 
unsolicited 

suggestions/ideas/alternatives 
provided  by  individual 

•  Self  ratings  on  participation 
(engagement,  alternatives 
generated) 

•  Participation  ratings  from  other 
pod  members  (engagement,  idea 
generation) 

•  time  coding  via 
audio/video  (post 
session) 

•  content  coding 
transcripts/audio/video 
(post  session) 

•  participation  ratings  - 
self  report  and  other 
report  (in  session  after 
each  task  if  possible 
(2X)  and  at  debrief) 

Will  need  to  develop 
in-house: 

•  process  for  time 
coding 

•  content  coding 
scheme 

•  engagement 

•  disengagement 

•  idea/alternative 
generation 

•  self  and  other 

measure 
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Variable 

Name 

Considerations 

Type  of  Measure  & 
Frequency 

Required  Actions 

Level  of 
participation 
(inter-pod) 

Would  be  rated  on  a  number  of 
components: 

•  Ratio  of  minutes  spent  on  intra¬ 
pod  communication  vs.  inter- pod 
communication 

•  Total  &  proportions  of  intra -pod 
communication  by  problem 
solving  stage  and  by  problem 
solving  approach. 

•  time  coding  via 
audio/video  (post 
session) 

Will  need  to  develop 
in-house: 

•  process  for  time 
coding 

Time  spent 
on  problem 
solving 
stages 

•  elapsed  time  (minutes)  and 
proportion  total  time  from  task 
start  to  production  of  problem 
definition  product 

•  elapsed  time  (minutes)  and 
proportion  total  time  from 
problem  definition  product  to 
solution  generation  product 

•  elapsed  time  (minutes)  and 
proportion  total  time  from 
solution  generation  product  to 
decision-making  product 

•  time  coding  via 
audio/video  (post 
session) 

Will  need  to  develop 
in-house: 

•  process  for  time 
coding 

•  demarcation  of 
product  completion 
criteria 

Pattern  of 
engagement 
in  problem 
solving 
stages 

•  pattern  of  time  spent  on  problem 
definition  activity  within  overall 
task  (minutes  distributed  across 
timeline) 

•  pattern  of  time  spent  on  solution 
generation  activity  within  overall 
task  (minutes  distributed  across 
timeline) 

•  pattern  of  time  spent  on  decision 
making  activity  within  overall 
task  (minutes  distributed  across 
timeline) 

•  proportion  of  inter-pod 
communication  by  problem 
solving  activity  type 

•  time  coding  via 
audio/video  (post 
session) 

•  content  coding 
transcripts/audio/video 
(post  session) 

Will  need  to  develop 
in-house: 

•  process  for  time 
coding 

•  content  coding 
scheme 

•  problem 
definition 
activity 

•  solution 
generation 
activity 

•  decision 
making 
activity 

Intra  pod 
task  and 
group 
cohesion 

Intra  pod  task  cohesion  would  be 

measured  on  concepts  of: 

•  degree  to  which  pod  was  task 
oriented  (Zaccaro,  1995) 

•  importance  that  the  pod  perform 
well  (Zaccaro,  1995) 

•  motivation  to  work  hard  so  pod 
succeeds  (Zaccaro,  1995) 

•  importance  that  other  members 
perform  well  on  the  task 
(Zaccaro,  1995) 

Intra  pod  group  cohesion  would  be 

measured  on  concepts  of: 

•  sense  of  belonging  to  pod  (Chin, 
1999) 

•  self  report  measures 
based  on  previous 
measures  in  this  area 

•  participation  ratings  - 
self  report  (in  session 
after  each  task  if 
possible  (2X)  and  at 
debrief) 

Will  need  to  adapt 
measures  for: 

•  task  cohesion 
(Zaccaro,  1995) 

•  group  cohesion 
(Chin,  1999) 
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Variable 

Name 

Considerations 

Type  of  Measure  & 
Frequency 

Required  Actions 

•  sense  of  pod  morale  (Chin,  1999) 

Inter  pod 
task  and 
group 
cohesion 

Task  cohesion  would  be  measured 

on  concepts  of: 

•  degree  to  which  group  of  pods 
were  task  oriented  (Zaccaro, 

1995) 

•  importance  that  the  group  of  pods 
perform  well  (Zaccaro,  1995) 

•  motivation  to  work  hard  so  the 
group  of  pods  succeed  (Zaccaro, 
1995) 

•  importance  that  other  pods 
perform  well  on  the  task 

Inter  pod  group  cohesion  (Zaccaro, 

1995) 

•  sense  of  belonging  to  group  of 
pods  (Chin,  1999) 

•  sense  of  group  of  pods  morale 
(Chin,  1999) 

•  self  report  measures 
based  on  previous 
measures  in  this  area 

•  participation  ratings  - 
self  report  (in  session 
after  each  task  if 
possible  (2X)  and  at 
debrief) 

Will  need  to  adapt 
measures  for: 

•  task  cohesion 
(Zaccaro,  1995) 

•  group  cohesion 
(Chin,  1999) 
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Problem  Solving  Outcomes 


Variable 

Name 

Considerations 

Type  of  Measure  & 
Frequency 

Required  Actions 

Decision 

quality 

Would  be  rated  on  a  number  of 

components  including  that  the 

decision  has  considered: 

•  impacts  of  event  on  multiple 
tiers 

•  impacts  of  implementing 
solutions  on  multiple  tiers 

•  psychosocial  implications  of 
event 

•  psychosocial  implications  of 
implementing  solutions 

•  general  resource 
constraints/availability 

•  specialized  resources  required 

•  time  requirements 

Would  be  rated  on  the  extent  to 

which  the  decision: 

•  reflects  multiple  perspectives 
from  within  the  pod 

•  reflects  multiple  perspectives 
from  outside  the  pod 

•  has  innovative/creative 
characteristics 

•  presents  a  cohesive  approach 
{not  just  every’  member 's 
solutions  piled  together) 

•  shares  information 

•  shares  resources 

•  shares  power/authority 

•  has  significant  operational 
gaps/challenges 

•  has  significant  strategic 
gaps/challenges 

•  content  coding 
transcripts  /  audio  / 
video  (post  session) 

•  judges’  ratings 

Will  need  to  develop  in- 

house: 

•  content  coding 
scheme  for  all 
dimensions  outlined 

•  template  for 
presenting  synopsis 
of  decision 

•  rating  guide  for 
judging  decision 
quality 

Satisfaction 
with  problem 
solving 
outcome 

•  Need  to  ensure  that  this  is 
separate  from  problem  solving 
process  -  emphasize  the 
outcome 

Key  components  would  include 

satisfaction  with: 

•  the  overall  quality  of  the 
outcome 

•  the  extent  to  which  outcome 
reflects  the  input  from  pod 
members 

•  the  likelihood  that  this  could  be 
implemented  successfully 

•  the  likelihood  that  this  would 
address  the  main  problems 

•  Self  report  (pen  and 
paper) 

•  Quick  measure  of 
key  components  after 
each  task  if  possible 
(2X)  (in-session) 

•  One  overall  for 
session  (debrief) 

Will  need  to  develop  in- 
house: 

•  Self  report  measure 
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Variable 

Name 

Considerations 

Type  of  Measure  & 
Frequency 

Required  Actions 

Level  of 
agreement 

Would  include: 

•  Self  rated  level  of  agreement 
with  outcome 

•  Observed  level  of  agreement 
among  pod  members 

•  Self  report  (pen  and 
paper) 

•  Quick  measure  of 
key  components  after 
each  task  if  possible 
(2X)  (in-session) 

•  One  overall  for 
session  (debrief) 

•  content  coding 
transcripts/audio/vid 
eo  (post  session) 

Will  need  to  develop  in- 
house: 

•  self  report  measure 

•  content  coding 
scheme  for  level  of 
agreement 

Changes  in 
individual 
organizational 
goals 

Would  include: 

•  Statement  of  individual 
organizational  goals  at  problem 
definition  stage 

•  Statement  of  individual 
organization  goals  post-task 

•  Statement  and 
restatement  of 
individual  goals 
would  need  to  be 
built  in  to  task 
instructions; 

•  content  coding  of 
task  outcomes  to 
determine  extent  to 
which  individual  org 
goals  changed 

Will  need  to  develop  in- 
house: 

•  tasks  that  can  include 
statements  of 
individual 

organization  goals  at 
the  problem 
definition  stage 

•  content  coding 
scheme  for  assessing 
goal  change 

Changes  in 
collective 
goals 

Would  include: 

•  Statement  of  collective  goals 
during  problem  definition  stage 

•  Statement  of  collective  goals 
post-task 

•  Statement  and 
restatement  of 
collective  goals 
would  need  to  be 
built  in  to  task 
instructions; 

•  content  coding  of 
task  outcomes  to 
determine  extent  to 
which  collective  org 
goals  changed 

Will  need  to  develop  in- 
house: 

•  tasks  that  can  include 
statements  of 
collective 

organization  goals  at 
the  problem 
definition  stage 

•  content  coding 
scheme  for  assessing 
goal  change 
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List  of  symbols/abbreviations/acronyms/initialisms 


ANCOVA 

Analysis  of  Covariance 

ANOVA 

Analysis  of  Variance 

CBRNe 

Chemical,  Biological,  Radiological,  Nuclear,  Explosive 

CEO 

Chief  Executive  Officer 

CF 

Canadian  Forces 

CORA 

Centre  for  Operational  Research  and  Analysis 

CRTI 

Chemical,  Biological,  Radiological,  Nuclear  and  Explosives  Research  ; 
Technology  Initiative 

CSS 

Centre  for  Security  Sciences 

D1 

Deliverable  1 

D2 

Deliverable  2 

D3 

Deliverable  3 

DRDC 

Defence  Research  &  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 
Management 

EM 

Emergency  Management 

EMO 

Emergency  Measures  Organization 

EOC 

Emergency  Operations  Centre 

FC 

Forces  canadiennes 

FEMA 

Federal  Emergency  Management  Agency 

GAP 

Groupe  d’ Analyse  Psychosociale 

I 

ICS 

IBM 

International  Business  Machines 

ICS 

Incident  Command  System 

IT 

Information  Technology 

M 

Military 

MANCOVA 

Multivariate  Analysis  of  Covariance 

MANOVA 

Multivariate  Analysis  of  Variance 

Ml 

Milestone  1 

M2 

Milestone  2 

M3 

Milestone  3 

MS 

Microsoft 

NGO 

Non-governmental  Organization 

NI 

Non-Incident  Command  System 

NIMS 

National  Incident  Management  System 

Non-ICS 

Non-Incident  Command  System 

ORG 

Organization 

PDF 

Portable  Document  Format 

PRiMer 

Psychosocial  Risk  Manager 

R&D 

Research  &  Development 

RCMP 

Royal  Canadian  Mounted  Police 

SARS 

Severe  Acute  Respiratory  Syndrome 

SDM 

Shared  Decision  Making 

TA 

Technical  Authority 

TIF 

Technology  Innovation  Fund 

TCPS 

Tri-Council  Policy  Statement 

WO 

Work  Order 

W1 

Work  Package  1 

W2 

Work  Package  2 

W3 

Work  Package  3 

W4 

Work  Package  4 

W5 

Work  Package  5 

W6 

Work  Package  6 
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